Automated Digitization of Unstructured Medical Prescriptions (2023)

Abstract: Automated digitization of prescription images is a critical prerequisite to scale digital healthcare services such as online pharmacies. This is challenging in emerging markets since prescriptions are not digitized at source and patients lack the medical expertise to interpret prescriptions to place orders. In this paper, we present prescription digitization system for online medicine ordering built with minimal supervision. Our system uses a modular pipeline comprising a mix of ML and rule-based components for (a) image to text extraction, (b) segmentation into blocks and medication items, (c) medication attribute extraction, (d) matching against medicine catalog, and (e) shopping cart building. Our approach efficiently utilizes multiple signals like layout, medical ontologies, and semantic embeddings via LayoutLMv2 model to yield substantial improvement relative to strong baselines on medication attribute extraction. Our pipeline achieves +5.9% gain in precision@3 and +5.6% in recall@3 over catalog-based fuzzy matching baseline for shopping cart building for printed prescriptions.

## introduction in recent years, prompted by the covid pandemic, there has been a rise in the adoption of online pharmaceutical services leading to improved access to medications and health outcomes. however, in emerging markets such as india, online pharmacy ordering continues to be challenging since prescriptions tend to be paper-based, unstructured and often, handwritten, which makes digitization a vital prerequisite. for in-store purchases, customers follow a simple process of presenting a prescription to the store pharmacist who interprets it and fulfills the order. current e-commerce purchase process, however, imposes a significant cognitive load on customers since they have to explicitly * equal contribution † work done while at amazon specify the medicines. this process is onerous for the customers due to their (a) unfamiliarity with the ordering process, (b) difficulty in understanding prescriptions, and (c) lack of expertise to interpret medical acronyms and identify substitute medicines. further, most online pharmacies have a post-cart creation workflow where customers upload the prescription to be verified by a remote pharmacist. lack of pharmacist capacity often leads to long wait time making the process unscalable. therefore, an automated system that converts prescription images to a digitized form to facilitate search-less shopping is essential for the success of online pharmacies. in particular, we need to extract the medical advice section which contains a list of medication items, each of which is a record of multiple fields such as brand-name. challenges. addressing this problem is non-trivial due to multiple reasons shown in figure 3a : (a) variability in prescription image quality, background, and orientation, (b) diversity of layouts and doctor styles, (c) high prevalence of typos that create confusion between similar items (e.g., fibrodone and firodone), (d) specialized vocabulary of regional prescriptions, and (e) the need for converting dosage-specific instructions to a precise product order. additionally, there are limited labeled prescriptions due to the high manual effort it entails. related work. while there have been significant advances in document ai @xcite @xcite and information extraction @xcite @xcite techniques, most of these methods are effective only on images of wellformatted documents such as invoices. besides, these generic methods require significant supervision and are not sufficiently modular to support a phased automation of the prescription processing workflow. recent work on digitizing medical prescriptions @xcite is focused on using named entity recognition (ner) methods for medication attribute detection, but these models perform poorly on non-us prescriptions due to vocabulary gaps and do not utilize the layout or catalog information. we present additional related work in appendix b. contributions. in this paper, we present a study on automated digitisation of prescription images with printed content which covers design, data, modeling and evaluation aspects. we discuss experimental results for our emerging modular pipeline which comprises a mix of ml and rule-based components for (a) image to text extraction and normalization (b) segmentation into blocks, medication items and extraction of medication attributes, (c) matching against pharmacy catalog, and (d) shopping cart building. we detail how our approach efficiently combines layout signal, medical ontologies, and semantic embeddings via layoutlmv2 model to yield substantial improvement relative to strong baselines on medication attribute extraction, and results in +5.9% and +5.6% gain in precision@3 and recall@3 over catalog-based fuzzy matching baseline for printed prescriptions. we discuss key learnings relevant for low data regime document ai systems in addition to presenting component-wise efficacy of our pipeline and results from ongoing experiments (appendix a.4.2) to highlight future directions. we present safety aspects in section 9. ## prescription digitization problem given a prescription image, a natural choice for digitization is in terms of conversion to a structured prescription object as per a global standard such as the fast healthcare interoperability resources (fhir) framework @xcite . since our objective is to create a shopping cart for automated medicine ordering we focus on populating the relevant fields only in the fhir prescription schema ( figure 1 depicts the various stages of processing a prescription (denoted by h a , h d , h o ) that results in the successive creation of following entities: (a) annotated prescription is a visually rich document (vrd) comprising labeled rectangular bounding boxes (bbs). each bb is associated with text and a list of annotations, which include the start-end offsets and labels corresponding to medication attributes, item boundaries, and block type, (b) digitized prescription is a structured object with canonical entries following the fhir-based prescription schema, (c) pharmacy order is a list of skus from the prescription along with the recommended quantities. the conversion to a pharmacy order (h o ) can be enabled via a deterministic lookup using a medicine-sku map if the medication codes in the digitized prescription are from the catalog. hence, we focus on the non-trivial transformations h a and h d that entail a data-driven approach. let @xmath0. for the i th prescription p i , let (a i , d i , o i ) denote the human annotated prescription, digitized prescription, and pharmacy order obtained from expert pharmacists, i.e., typically, pharmacists directly create or validate pharmacy orders from a prescription image without any record of the intermediate annotation and digitization. since these prescription-order pairs are inadequate for an end-to-end neural model, we explicitly gather supervision for the intermediate stages for a subset of the prescriptions to enable a pipelined approach. let z a i , z d i , z o i denote binary indicators of the availability of a i , d i , o i respectively. further l a (•, •), l d (•, •), l o (•, •) be suitable loss functions for comparison of pairs of candidate annotated versions, digitized versions, and orders corresponding to a prescription such as the accuracy of annotations, matching with canonical entities, and the constructed order respectively as shown in given a new prescription, the learned mappings ( ĥa , ĥd ) along with h o yield a pharmacy order. ## data collection due to the sensitive nature of prescriptions and recent emergence of our medicine ordering application, a public dataset of unstructured prescription images does not exist to the best of our knowledge. external benchmarks such as @xcite only contain clean text without any layout information. hence, we use a proprietary dataset of 1359 indian prescriptions paired with (fully or partial) digitized orders, delinked from customer ids. these are mostly or fully printed prescriptions and have been validated by our in-house experts. the prescription images are modified as follows prior to modeling. aws textract, which is security certified for critical data, is used to extract text from images. the obtained text is then run through aws comprehend to detect personally identifiable information such as patient/doctor names, phone numbers and then the corresponding ocr bounding boxes are grayed out. for a subset of prescriptions, we procured in-house human annotations for supervised training of all components. ground-truth text, bbs for medical advice blocks, labeled text spans for medication attributes, as well labels for pairs of candidate and ground truth skus for medication matching were annotated in the prescription image by the annotators. more details on the annotation tasks are given in ## experimental results we present our evaluation method and results on the efficacy of the full system and various components with focus on medication attribute extraction. ## concluding remarks prescription digitization is a critical enabler of online pharmacy services. we present a holistic, modular approach to address this problem in a low data regime using hybrid ml and rule-based components. our approach uses layout signals, medical ontologies, sequential dependencies, and semantic embeddings to yield significant improvement over baselines and good performance on unstructured printed prescriptions. ongoing directions include using active learning to judiciously label data (section a.4.2), pseudo labeling of partially digitized orders and digitizing handwritten prescriptions.