Is That the Right Dose? Investigating Generative Language Model Performance on Veterinary Prescription Text Analysis (2024)

Abstract: Optimizing antibiotic dosing recommendations is a vital aspect of antimicrobial stewardship (AMS) programs aimed at combating antimicrobial resistance (AMR), a significant public health concern, where inappropriate dosing contributes to the selection of AMR pathogens. A key challenge is the extraction of dosing information, which is embedded in free-text clinical records and necessitates numerical transformations. This paper assesses the utility of Large Language Models (LLMs) in extracting essential prescription attributes such as dose, duration, active ingredient, and indication. We evaluate methods to optimize LLMs on this task against a baseline BERT-based ensemble model. Our findings reveal that LLMs can achieve exceptional accuracy by combining probabilistic predictions with deterministic calculations, enforced through functional prompting, to ensure data types and execute necessary arithmetic. This research demonstrates new prospects for automating aspects of AMS when no training data is available.

## introduction antimicrobial resistance (amr) has become a major public health concern, as antimicrobials are steadily losing their effectiveness in combating bacterial infections @xcite . amr is not limited to human medicine; it is also a growing issue among animals @xcite , who can acquire and transmit multidrug-resistant pathogens to humans @xcite . antimicrobial stewardship (ams), which has demonstrated effectiveness in improving antimicrobial use in both human and animal healthcare @xcite , aims to optimize antimicrobial use to curtail the development and spread of amr. accurate dosing is part of this strategy, as overdosing can lead to toxicity and under-dosing can be particularly perilous as it can select for amr organisms and lead to poor therapeutic outcomes @xcite . a pragmatic way to improve dosing accuracy and optimizing antimicrobial use is through decision support systems in clinical settings @xcite , where targeted dosing recommendations can be made in real-time. recent developments in large language models (llms) introduce compelling opportunities for automated information extraction and decision support @xcite , as these models obviate the need for extensive labeled data @xcite . such models can potentially furnish clinicians with data-driven counsel on optimal antimicrobial selection, treatment du-ration, and dosing intervals, tasks that have been historically reliant on extensive labor-intensive labeled data compilation @xcite . to realize the potential for llms for extracting prescription elements, an essential step is empirical assessment of their ability to accurately extract relevant information from clinical text. given the idiosyncratic nature of llm training, which leverages instruction tuning rather than conventional training paradigms, it becomes vital to also scrutinize configuration variances for performance optimization @xcite . additionally, while the task of extracting elements out of prescriptions was explored in shared tasks such as the 2010 i2b2 challenge @xcite , these studies only evaluate the ability to extract text spans without performing numerical conversion. converting text spans to numerical representations and performing necessary calculations to understand the dose and duration of a given medication are also essential to optimize antimicrobial use. studies performing such numerical conversions rely on rules-based methods which are notoriously brittle, and only one study we identified made the corresponding algorithm available @xcite . we leverage the vetcompass australia @xcite corpus, which comprises over 50 million clinical notes from over 200 veterinary clinics across australia, as our primary data source. our goal is to extract key information such as the active ingredient, the indication for antimicrobial use, and the dose and duration of the therapy. we assess the performance of llms in zero-shot and few-shot learning scenarios for extracting this critical information. by exploring the feasibility of applying llms to the vetcompass dataset, we seek to understand their potential in aiding dosing recommendations to support ams. specifically: we investigate the task of dose information extraction from veterinary clinical notes. given textual clinical notes, the task is to extract seven labels, including five entity labels: active ingredient, clinical indication, frequency, medication size, and dosage unit size, along with two derived labels: dose and duration. a sample clinical note, prescription label, and the inferred target labels are illustrated in figure 1. prescription label information is provided as an input for all extractions except indication, for reasons of document length. for indication, the model is also given the set of potential indications; for ingredient, the set of potential ingredients. we evaluate accuracy based on exact match between the output label and the ground truth label. ## methodology we benchmark three llms on this task: gpt-3.5 (brown et al., 2020), gpt-4 (openai, 2023a), and llama2-70b @xcite , against the baseline ensemble model (rxvetbert), which combines rule-based methods and vetbert as described in previous works @xcite . prompt settings we compare the following: zero-shot: utilizes text from the clinical note and/or prescription label, along with the item name, weight of the patient, and the number of units dispensed as input. a prompt for the element being classified is included. no examples are provided. few-shot random examples: incorporates randomly-sampled example prescriptions or examination text and inferred labels from the training set as in-context examples @xcite . to manage token limits, we include three labeled examples for prescription prompts and two for indication prompts-the examination text required for the indication prompt were much longer. few-shot similar examples: instead of random examples, we use text similarity as a selection criterion to retrieve examples for in-context learning @xcite @xcite . for the retriever, we employ a distilled sbert model @xcite to encode text and retrieve examples based on cosine similarity. ## results and discussion overall, our experiments find that llms are highly effective at extracting and interpreting numerical elements (e.g., frequency and dose unit size) necessary to calculate the dose and duration ( refining the labeling schema for indications is a promising avenue for mitigating this issue. conclusion this paper provides a framework for llms to extract essential prescription data from veterinary text, such as dose, duration, and active ingredients for supporting ams efforts. we overcome limitations in calculating elements such as dose and duration by integrating probabilistic outputs with deterministic calculations through functional prompting, even in zero-shot settings. future work should consider evaluation for human clinical applications, given the potential contributions of this approach to broader healthcare.