R x L ens: Multi-Agent LLM -powered Scan and Order for Pharmacy (2025)

Abstract: Automated construction of shopping cart frommedical prescriptions is a vital prerequisite forscaling up online pharmaceutical servicesin emerging markets due to the high prevalence of paper prescriptionsthat are challenging for customers to interpret.We present RxLens, a multi-step end-end Large Language Model (LLM)-based deployed solutionfor automated pharmacy cart construction comprisingmultiple steps: redaction of Personal Identifiable Information (PII),Optical Character Recognition (OCR), medication extraction, matching against the catalog, and bounding box detection for lineage. Our multi-step design leverages the synergy between retrieval and LLM-based generationto mitigate the vocabulary gaps in LLMs and fuzzy matching errors during retrieval.Empirical evaluation demonstrates that RxLens can yield up to 19% - 40% and 11% - 26% increase in Recall@3 relative to SOTA methods such as Medical Comprehend and vanilla retrieval augmentation of LLMs on handwritten and printed prescriptions respectively.We also explore LLM-based auto-evaluation as an alternative to costly manual annotations and observe a 76% - 100% match relative to human judgements on various tasks.

## introduction global adoption of online pharmacy services has surged in recent years, driven by demand for convenient, affordable access to medications. however, in emerging markets, paper prescriptions, which are typically unstructured, handwritten, and illegible, pose a major barrier for customers ordering medications online. patients often report difficulties in deciphering doctors' handwriting accurately enough to use traditional e-commerce search. to mitigate the digitization errors and the consequent health risks, e-pharmacies offer "medicine dispensation" services where customers can upload prescriptions and receive cart-building assistance through either asynchronous digitization or direct pharmacist callbacks. while pharmacist calls provide better accuracy and capture specific needs like medication quantities and alternatives, they are costlier. both approaches face scalability challenges due to the reliance on human pharmacists, resulting in long wait times and high cart abandonment. hence, there is an urgent need for an automated, rapid, accurate, and scalable prescription digitization system to enable seamless online pharmacy ordering. building automated prescription-to-cart systems poses several key challenges. these span handling diverse layouts and handwriting styles, varying image quality and orientation, and region-specific medical terminology. further, typos frequently cause confusion between similar drug names, making high accuracy critical for patient safety. a practical system must also secure patient pii while precisely mapping medications to the visual region on prescriptions. lastly, the sensitive nature of prescriptions combined with expensive annotation effort leads to a significant scarcity of ground truth, complicating system development and evaluation. related work. current prescription digitization methods @xcite ) follow a multi-step process: (a) optical character recognition, (b) medication extraction using custom-trained text and/or layout encoder models, and (c) matching extracted medications against a catalog. these methods perform poorly on non-us and handwritten prescriptions due to vocabulary gaps and limited training data. studies on handwritten prescriptions @xcite @xcite have achieved limited success in identifying medicine names. despite the broad success of recent foundational generative llms and multimodal approaches @xcite , their adoption for prescription digitization remains minimal. these models, trained primarily on public datasets with limited handwritten documents and regional medical vocabulary, fail to achieve the desired accuracy when used directly or with vanilla retrieval augmentation, often due to hallucination. further, the high llm deployment costs @xcite and pii concerns with third-party llm apis complicate their use in prescription digitization. appendix a presents additional related work. contributions. we explore how to use llms (both multimodal and text-only) to develop an automated prescription-to-cart system. we investigate choices related to solution architecture, componentspecific design, annotation scaling, and practical deployment, and present the below key contributions: 1) building on existing methods, we propose rxlens, a modular llm-based architecture comprising ocr, medication extraction, and matching against the catalog. this multi-step design leverages catalog-based retrieval augmentation to ensure medication validity. within each step, we explore the benefits of llms and prompting strategies, focusing on the synergy between retrieval and generation. 2) we present solutions for handling practical system requirements, such as pii redaction before llm invocation, medication-toprescription region mapping, and latency optimization. 3) to address the lack of annotations and expensive labeling, we develop an llm-based autoevaluation approach using prompts that mimic human annotation (75.7% -100% correlation). 4) empirical evaluation shows rxlens achieves significant improvements (+19%-40% and +11%-26% recall@3) over sota baselines like medical comprehend and vanilla llm retrieval augmentation on handwritten and printed prescriptions, respectively. ## prescription image digitization formally, given a medicine catalog a 1 , a prescription image p , and k, the max. number ## experimental setup we describe our setup for evaluating llm-based prescription digitization focusing on questions related to solution architecture, component choices, deployment constraints, and auto-evaluation. ## conclusion and future work our current work presents an llm-based architecture of a deployed system for digitizing medical prescriptions, assessing various design choices including data privacy and usability. summary of key learnings. 1) specialized models can sometimes outperform foundational models, such as textract trained on document images outperforming claude. 2) retrieval augmentation with relevant context can yield significant performance benefits for specialized domains like pharmacy. 3) reranking with llms improves top ranking results due to their intrinsic world knowledge and ability to perform fuzzy matching over textual attributes. 4) auto-evaluation using llms closely matches human evaluation, enabling scalable monitoring and system optimization. 5) for real-time applications, latency is an important factor, making it crucial to focus on parallelization opportunities. future directions. we also plan to explore (a) specialized multimodal models for handwritten content recognition, (b) automated prompt optimization using meta-prompting strategies, (c) assess-ment of auto-evaluation with more manual annotations. the approach can also be extended to digitizing other documents such as shopping lists.