Zero-Shot ATC Coding with Large Language Models for Clinical Assessments (2025)

Abstract: Manual assignment of Anatomical Therapeutic Chemical (ATC) codes to prescription records is a significant bottleneck in healthcare research and operations at Ontario Health and InterRAI Canada, requiring extensive expert time and effort. To automate this process while maintaining data privacy, we develop a practical approach using locally deployable large language models (LLMs). Inspired by recent advances in automatic International Classification of Diseases (ICD) coding, our method frames ATC coding as a hierarchical information extraction task, guiding LLMs through the ATC ontology level by level. We evaluate our approach using GPT-4o as an accuracy ceiling and focus development on open-source Llama models suitable for privacy-sensitive deployment. Testing across Health Canada drug product data, the RABBITS benchmark, and real clinical notes from Ontario Health, our method achieves 78% exact match accuracy with GPT-4o and 60% with Llama 3.1 70B. We investigate knowledge grounding through drug definitions, finding modest improvements in accuracy. Further, we show that fine-tuned Llama 3.1 8B matches zero-shot Llama 3.1 70B accuracy, suggesting that effective ATC coding is feasible with smaller models. Our results demonstrate the feasibility of automatic ATC coding in privacy-sensitive healthcare environments, providing a foundation for future deployments.

## introduction the anatomical therapeutic chemical (atc) classification system is a standardized drug ontology maintained by the world health organization (who). assigning atc codes to drug mentions is essential for various healthcare operations, including medication inventory management, drug utilization research, and health insurance claims processing. however, manual atc coding is timeconsuming and requires expert knowledge, creating a significant bottleneck in healthcare workflows. our work represents a collaboration between computer scientists and public health researchers at interrai, foot_0 aimed at addressing this critical workflow challenge. interrai canada receives assessment data from ontario health, @xcite where clinical experts must manually review each prescription record and assign appropriate atc codes before any population-level analysis can begin. this manual process substantially delays both operational reporting and critical public health research, particularly studies on drug utilization patterns and medical practice variations across care facilities. the challenge is particularly acute in processing unstructured clinical text, where drug mentions may appear as brand names, generic names, or various informal descriptions. while recent advances in large language models (llms) have shown promise in medical coding tasks, deploying these solutions in healthcare settings raises important privacy concerns. many state-of-the-art models require data to be sent to proprietary apis, making them unsuitable for handling sensitive clinical information. to address these challenges, we present a practical approach to automatic atc coding designed specifically for deployment in privacy-sensitive healthcare environments. our method frames atc coding as a hierarchical information extraction task, leveraging open-source llms to navigate the atc ontology level by level. we evaluate our approach against gpt-4o as an accuracy ceiling while focusing development on locally deployable llama models, making a first attempt at automatic atc coding with llms. in developing this solution for public health researchers at ontario health and interrai canada, we make several key contributions: our results demonstrate the viability of automated atc coding in real-world healthcare settings while highlighting important considerations for deploying llm-based solutions in privacy-sensitive environments. this work provides a foundation for healthcare researchers and organizations seeking to automate their coding processes without compromising data privacy or security. the atc ontology. the atc classification system is the global standard for drug classification maintained by the who. it organizes drugs into a five-level hierarchical structure based on the organ system they target and their therapeutic, pharmacological, and chemical properties. each atc code consists of seven characters encoding these five levels: icd coding, the atc classification system has received comparatively little attention in the context of language model applications. current atc coding practices rely predominantly on three approaches: manual coding by clinical experts, rulebased systems utilizing string matching against generic drug names, or hybrid systems combining both approaches (pang et al., 2015; kellmann et al., 2023). these existing methods, particularly the manual processes, are not only time-intensive but also susceptible to human error, highlighting the need for more efficient and accurate solutions. to address this gap in the literature, our study presents the first investigation into utilizing llms for automatic atc coding, to the best of our knowledge. ## results and discussion zero-shot effectiveness. for the ontario health assessments, among the 200 clinical prescription notes, 166 (83%) are assigned granularity level 5, indicating that a precise level-5 atc code can be deduced with confidence. the remaining 34 notes are distributed across other granularity levels: 20 at level 0, 0 at level 1, 2 at level 2, 2 at level 3, and 10 at level 4. we evaluate the open-source llama 3.1 models (excluding gpt-4o due to privacy constraints) on the 166 unambiguous samples. the results are on par with the health canada product names, particularly at correct @xmath0. this validates our hypothesis that real-world drug prescriptions are often variations of product names, and suggests that gpt-4o would likely achieve similar zero-shot effectiveness on the ontario health assessments as observed with the health canada product names. fine-tuning effectiveness notably, when finetuned on the health canada and rabbits training sets, llama 3.1 8b consistently surpasses the zeroshot accuracy of the larger llama 3.1 70b model across all three datasets. this demonstrates that effective atc coding is possible with smaller, locally deployable models when task-specific training data is available. knowledge grounding effectiveness. ## conclusion in this work, we present a practical approach to automatic atc coding using llms, demonstrating meaningful zero-shot effectiveness on both curated product-name datasets and real clinical prescriptions. further, we show that fine-tuned smaller models can achieve comparable effectiveness, showcasing the potential of automated atc coding with limited computational resources. our analysis reveals several important insights for real-world deployment. first, the similarity in effectiveness between ontario health prescriptions and health canada product names suggests that drug mentions in prescription settings often appear as variations of product names, where our approach demonstrates strong zero-shot accuracy. second, our investigation of knowledge grounding demonstrates that while additional context can improve fine-grained classification at deeper levels, the improvements are modest overall. finally, the effectiveness gap between health canada and rabbits datasets highlights a key limitation: current llms rely heavily on string similarity between product names and their generic counterparts, suggesting an area for future improvement. looking ahead, in addition to addressing the string similarity challenge, several directions could enhance the practical utility of our system. developing more efficient knowledge grounding strategies could improve accuracy without sacrificing speed, and exploring hybrid approaches that combine llm-based classification with traditional rulebased systems might provide more robust solutions for healthcare organizations. to conclude, our work demonstrates the feasibility of automated atc coding with llms, while also setting the groundwork for building careful systems that balance accuracy, privacy, and computational requirements in healthcare settings.