Retrieve, Reason, and Refine: Generating Accurate and Faithful Patient Instructions
Abstract
The “Patient Instruction” (PI), which contains critical instructional information provided both to carers and to the patient at the time of discharge, is essential for the patient to manage their condition outside hospital.
An accurate and easy-tofollow PI can improve the self-management of patients which can in turn reduce hospital readmission rates.
However, writing an appropriate PI can be extremely time-consuming for physicians, and is subject to being incomplete or error-prone for (potentially overworked) physicians.
Therefore, we propose a new task that can provide an objective means of avoiding incompleteness, while reducing clinical workload: the automatic generation of the PI, which is imagined as being a document that the clinician can review, modify, and approve as necessary (rather than taking the human “out of the loop”).
We build a benchmark clinical dataset and propose the Re 3 Writer, which imitates the working patterns of physicians to first re trieve related working experience from historical PIs written by physicians, then re ason related medical knowledge.
Finally, it re fines the retrieved working experience and reasoned medical knowledge to extract useful information, which is used to generate the PI for previously-unseen patient according to their health records during hospitalization.
Our experiments show that, using our method, the performance of five different models can be substantially boosted across all metrics, with up to 20%, 11% and 19% relative improvements in BLEU-4, ROUGE-L and
METEOR, respectively.
Meanwhile, we show results from human evaluations to measure the effectiveness in terms of its usefulness for clinical practice.
Introduction
At the time of discharge to home, the Patient Instruction (PI), which is a rich paragraph of text containing multiple instructions, is provided by the attending clinician to the patient or guardian. PI
[Table] [Table] is used for the purpose of facilitating safe and appropriate continuity of care [7, 38, 44].
As a result, the PI has significant implications for patient management and good medical care, lowering down the risks of hospital readmission, and improving the doctor-patient relationship.
As shown in Figure 1, a
PI typically contains the following three main components from patients’ perspective [18, 47]: (1)
What is my main health condition?
(i.e., why was I in the hospital?)
(2) What do I need to do?
(i.e., how do I manage at home, and how should I best care for myself?)
(3) Why is it important for me to do this?
Of the above, the second section is often considered to be the most important information.
For example, when a patient has had surgery while in the hospital, the PI might tell the patient to keep the wound away from water to avoid infection.
Currently, the following skills are needed for physicians to write a PI [38, 44, 7]: (1) thorough medical knowledge for interpreting the patient’s clinical records, including diagnosis, medication, and procedure records; (2) skills for carefully analyzing the extensive and complex patient’s data acquired during hospitalization, e.g., admission notes, nursing notes, radiology notes, physician notes, medications, and laboratory results; (3) the ability of extracting key information and writing instructions appropriate for the lay reader.
Therefore, writing PIs is a necessary but time-consuming task for physicians, exacerbating the workload of clinicians who would otherwise focus on patient care [52, 42].
Besides, physicians need to read lots of patient’s health records in their daily work, resulting in substantial opportunity for incompleteness or inappropriateness of wording [46, 53].
Statistically, countries with abundant healthcare resources, such as the United States, have up to 54% of physicians experiencing some sign of burnout in one year of study [43], which is further exacerbated in countries with more tightly resource-constrained healthcare resources.
The overloading of physicians is a well-documented phenomenon [46, 53], and AI-related support systems that can partly automate routine tasks, such as generation of PIs for modification/approval by clinicians is an important contribution to healthcare practice.
To this end, we propose the novel task of automatic PI generation, which aims to generate an accurate and fluent textual PI given input health records of a previously-unseen patient during hospitalization.
In this way, it is intended that physicians, given the health records of a new patient, need only review and modify the generated
PI, rather than writing a new PI from scratch, significantly relieving the physicians from the heavy workload and increasing their time and energy spent in meaningful clinical interactions with patients.
Such endeavors would be particularly useful in resource-limited countries [48].
In this paper, we build a dataset PI and propose a deep-learning approach named Re 3 Writer, which imitates the physicians’ working patterns to automatically generate a PI at the point of discharge from hospital.
Specifically, when a patient discharges from the hospital, physicians will carefully analyze the patient’s health records in terms of diagnosis, medication, and procedure, then accurately write a corresponding PI based on their working experience and medical knowledge [6, 7].
In order to model clinicians’ text production, the Re 3 Writer, which introduces three components: Re trieve, Re ason, and Re fine, (1) first encodes working experience by mining historical PIs, i.e., retrieving instructions of previous patients according to the similarity of diagnosis, medication and procedure; (2) then reasons medical knowledge into the current input patient data by learning a knowledge graph, which is constructed to model domain-specific knowledge structure; (3) at last, refines relevant information from the retrieved working experience and reasoned medical knowledge to generate final PIs.
To prove the effectiveness of our Re 3 Writer, we incorporate it into 5 different language generation models: 1) recurrent neural network (RNN)-based model, 2) attention-based model, 3) hierarchical
RNN-based model, 4) copy mechanism-based model, 5) fully-attentive model, i.e., the Transformer [49].
The extensive experiments show that our approach can substantially boost the performance of baselines across all metrics.
Overall, the main contributions of this paper are:
• We make the first attempt to automatically generate the Patient Instruction (PI), which can reduce the workload of physicians.
As a result, it can increase their time and energy spent in meaningful interactions with patients, providing high-quality care for patients and improving doctor-patient relationships.
• To address the task, we build a dataset PI and propose an approach Re 3 Writer, which imitates physicians’ working patterns to retrieve working experience and reason medical knowledge, and finally refine them to generate accurate and faithful patient instructions.
• We prove the effectiveness and the generalization capabilities of our approach on the built PI dataset.
After including our approach, performances of the baseline models improve significantly on all metrics, with up to 20%, 11%, and 19% relative improvements in BLEU4, ROUGE-L, and METEOR, respectively.
Moreover, we conduct human evaluations to the generated PI for its quality and usefulness in clinical practice.
Related Works
The related works are introduced from two aspects: 1) Natural Language Generation (NLG) and 2)
Medical Text Generation.
Natural Language Generation (NLG)
It aims to automatically generate coherent text in naturallanguage from given corresponding input data, which can be, e.g., text [2], images [51], video [50], and audio [9].
Commonly, approaches use an encoder-decoder framework, where the encoder computes intermediate representations for the input source data and the decoder adopts RNNs or
CNNs [16, 10], to generate the target sentences given the intermediate representation.
The attention mechanism [34, 49, 54] and the copy mechanism [12] have been proposed to directly provide the decoder with the source information, enabling a more efficient use of the source data.
In particular, fully-attentive models, such as the Transformer [49], in which no recurrence is required, have successfully achieved state-of-the-art performance for multiple natural language generation tasks.
However, most existing work focuses on the natural text-related data, where research concerning the clinical domain remains relatively under-studied.
Medical Text Generation
Recently, language generation in the medical domain has received growing research interest.
For example, Jing et al.
[20], Liu et al.
[30, 31] proposed to generate radiology reports from chest X-ray images; Lee [25] proposed to generate the clinical notes of emergency department cases from discharge diagnosis codes; Ive et al.
[19] proposed to generate mental health records.
Meanwhile, some works treated the generation of medical text as a summarization task.
For example, Zhang et al.
[55] proposed to generate clinical impressions by summarizing radiology reports; Scott et al.
[40] proposed to generate the patients’ histories by summarizing their respective health records; Several works [37, 13, 33, 11, 41], including HARVEST [15] - a medical summarization tool, proposed to summarize patients’ key conditions from their clinical health records, such as Hassanpour and Langlotz [13] and Shing et al.
[41] which summarized important clinical entities from radiology reports and hospital health records, respectively.
Despite the encouraging performances of these methods in generating various types of medical text, to the best of our knowledge, none of them has attempted to generate the Patient Instructions, which has significant implications for supporting care delivery [7, 38, 44] as noted earlier.
We here make the first attempt to generate the Patient Instructions, which is typically longer than existing NLG-related output in the medical domain and which covers more diverse topics than previous medical text generation tasks, such as clinical impressions [55], clinical notes [25], and radiology reports [29, 32].
Input (Patient's Health Records)
Nursing Notes
Radiology Notes
Physician Notes
Medications Admission Notes … …
Multi-HeadAttention
Add & Norm Add & Norm
FeedForward
Word Embedding
Record Encoder
Multi-HeadAttention
Add & Norm Add & Norm
FeedForward
Reason
Add & Norm
Linear & Softmax
PI Decoder
Working Experience
Medical Knowledge
Retrieved PI of Patient 3 ∗𝑁𝑁 P
Retrieved PI of Patient
Retrieved PI of Patient… …
Target Instruction 𝐺𝐺 Pr 𝑅𝑅 ′ 𝑊𝑊
Pr
Output Probabilities
Refine
Retrieve
Approach
We first define the PI generation problem; then, we describe the proposed Re 3 Writer in detail. 3.1 PI Generation Problem Definition When a patient is discharged from the hospital, the PI generation system should generate a fluent and faithful instruction to help the patient or carer to manage their conditions at home. Therefore, the goal of the PI generation task is to generate a target instruction I = { y 1 , y 2 , . . .
, y N I } given the patient’s health records R = { r 1 , r 2 , . . .
, r N R } in terms of diagnoses, medications and procedures performed during hospitalization.
Since the input R including N R words and the output I including N I words are both textual sequences, we adopt the encoder-decoder framework, which is widely-used in natural language generation tasks, to perform the PI generation task.
In particular, the encoder-decoder framework includes a health record encoder and a PI decoder, which can be formulated as:
[Equation]
where R ′ ∈
R N R × d denotes the record embeddings encoded by the record encoder, e.g., LSTM [16] or Transformer [49].
Then, R ′ is fed into the PI decoder (which again could be an LSTM or
Transformer), to generate the target PI I .
During training, given the ground truth PI for the input patient’s health records, we can train the model by minimizing the widely-used cross-entropy loss.
3.2 The Proposed Re 3 Writer
Our Re 3 Writer consists of three core components: Retrieve, Reason, and Refine.
Formulation of the Re 3 Writer
As stated above, given the health records R encoded as R ′ ∈
R N R ×
d , we aim to generate a desirable PI I .
Figure 2 shows the detail of our method, which is designed to retrieve related working experience W Pr and reason related medical knowledge G Pr from the training corpus for current input patient.
Finally, Re 3 Writer refines the retrieved working experience W Pr and reasoned medical knowledge G Pr to extract useful information to generate a proper PI:
Record Encoder : R → R ′ ;
Retrieve : R → W Pr ; Reason : R → G Pr ;
Refine + PI Decoder : { R ′ , W Pr , G Pr } → I. [Equation]
Our method can be incorporated into existing encoder-decoder based models, to boost their performance with PI generation, as we will later demonstrate.
We now describe how to retrieve related working experience and reason related medical knowledge from the training corpus for PI generation.
Retrieve
As shown in Figure 1, a hospitalization typically produces records of diagnoses given, medications used, and procedures performed; our dataset (described later) has 11,208 unique clinical codes, including 5,973 diagnosis codes, 3,435 medication codes, and 1,800 procedure codes.
Therefore, we first extract the one-hot embeddings of all clinical codes.
Given a new patient, we represent this patient’s hospitalization by averaging the associated one-hot embeddings of clinical codes produced during this hospitalization.
Then, we collect a set of patients similar to the new patient according to the associated clinical codes.
Taking the diagnosis codes as an example, we retrieve N P patients from the training corpus with the highest cosine similarity to the input diagnosis codes of the current patient.
The PIs of the top- N P retrieved patients are returned and encoded by a
BERT encoder [8, 39], followed by a max-pooling layer over all output vectors, and projected to d dimensions, generating the related working experience in terms of Diagnosis for the current patient: { I 1 , I 2 , . .
, I N P } ∈ R N P × d .
Similarly, we use the medication codes and procedure codes to acquire the working experience in terms of Medication and Procedure, respectively.
At last, we concatenate these code-specific working experience representation (Diagnosis, Medication, Procedure) to obtain the final working experience related to current patient W Pr
= { I 1 , I 2 , . .
, I 3 ∗ N P } ∈ R (3 ∗ N P ) × d .
We also attempted to incorporate age and gender information into our approach to match patients, please see Appendix C for details.
Reason
To reason related medical knowledge from the training corpus, we construct an off-the-shelf global medical knowledge graph G
=
( V, E ) using all clinical codes, i.e., diagnosis, medication, and procedure codes, across all hospitalizations, where V = { v i } i =1: N KG ∈ R N KG
×
d is a set of N KG nodes and E = { e i,j } i,j =1: N KG is a set of edges.
The G models the domain-specific knowledge structure.
In detail, we consider the clinical codes as nodes.
The weights of the edges are calculated by normalizing the co-occurrence of pairs of nodes in the training corpus.
After that, guided by current input patient’s health records, the knowledge graph is embedded by a graph convolution network (GCN) [35, 26, 24], acquiring a set of node embeddings { v ′ 1 , v ′ 2 , . . .
, v ′ N KG } , which is regarded as our reasoned medical knowledge G Pr
= { v ′ 1 , v ′ 2 , . .
. , v ′ N KG } ∈ R N KG × d .
Please refer to Appendix A for the detailed description of our medical knowledge graph.
We note that more complex graph structures could be constructed by using larger-scale well-defined medical ontologies.
Therefore, our approach is not limited to the currently constructed graph and could provide a good basis for future research in this direction.
Refine
As shown in Eq.
(2), the PI decoder equipped with our method aims to generate the final
PI based on the encoded patient’s health records R ′ ∈
R N R × d , the retrieved working experience
W Pr ∈ R (3 ∗ N P ) ×
d and the reasoned medical knowledge G Pr ∈ R N KG × d .
In implementations, we can choose either the LSTM [16] or Transformer [49] as the decoder.
Taking the Transformer decoder as an example: for each decoding step t , the decoder takes the embedding of the current input word x t = w t + e t ∈ R d as input, where w t and e t denote the word embedding and fixed position embedding, respectively; we then generate each word y t in the target instruction I = {
[Equation]
where the MHA and FFN respectively stand for the Multi-Head Attention and Feed-Forward Network in the original Transformer (see Appendix D);
W p ∈ R
d ×| D | and b p are the learnable parameters ( | D | : vocabulary size); the Refine component then refines the W Pr and G Pr to extract useful and correlated information to generate an accurate and faithful PI.
Intuitively, the PI generation task aims to produce an instruction based on the source patient’s health records R ′ , supported with appropriate working experience W Pr and medical knowledge G Pr .
Thus,
W Pr and G Pr play an auxiliary role during the PI generation.
To this end, the Refine component, which makes the model adaptively learn to refine correlated information, is designed as follows: Refine ( h t ,
[Equation]
[Equation]
where W h 1 , W h 2 ∈ R 2 d × d and b h 1 , b h 2 are learnable parameters.
⊙ , σ , and [ · ; · ] denote the elementwise multiplication, the sigmoid function, and the concatenation operation, respectively.
The computed λ 1 , λ 2 ∈ [0 , 1] weight the expected importance of W Pr and G Pr for each target word.
In particular, if LSTM is adopted as the PI decoder, we can directly replace the MHA with the LSTM unit and remove the FFN in Eq.
In the following sections, we will prove that our proposed
Re 3 Writer can provide a solid basis for patient instruction generation.
Experiment
We first introduce our dataset used as the basis for our experiments, as well as the baseline models and settings.
We subsequently show evaluations using both automatic and “human” approaches.
4.1 Dataset, Baselines, and Settings
Dataset We propose a benchmark clinical dataset Patient Instruction (PI) with around 35k pairs of input patient’s health records and output patient instructions.
In detail, we collect the PI dataset from the publicly-accessible MIMIC-III v1.4 resource 4 [22, 21], which integrates de-identified, comprehensive clinical data for patients admitted to the Intensive Care Unit of the Beth Israel
Deaconess Medical Center in Boston, Massachusetts, USA.
This resource is an important “benchmark” dataset that promotes easy comparison between studies in the area.
For each patient in the MIMIC-III v1.4 resource, the dataset includes various patient’s health records during hospitalization in terms of diagnoses, medications and procedures, e.g., demographics, laboratory results, admission notes, nursing notes, radiology notes, and physician notes. In our experiments, we found that the discharge summaries contain the abstractive information of patient’s health records.
Therefore, for clarity, we adopt such abstractive information to generate the PI.
We concatenate all available patients’ health records as the input of our model. [Table]
For data preparation, we first exclude entries without a patient instruction and entries where the word-counts of patient instructions are less than five.
This results in our PI dataset of 28,029 unique patients and 35,851 pairs of health records and patient instructions, as summarized in Table 1.
We randomly partition the dataset into 80%-10%- 10% train-validation-test partitions according to patients.
Therefore, there is no overlap of patients between train, validation and test sets.
Next, we pre-process the records and instructions by tokenizing and converting text to lower-case.
Finally, we filter tokens that occur fewer than 20 times in the corpus, resulting in a vocabulary of approximately 19.9k tokens, which covers over 99.5% word occurrences in the dataset.
Baselines We choose five representative language generation models with different structures as baseline models, i.e., 1) RNN-based model ( LSTM ) [5], 2) attention-based model ( Seq2Seq ) [2, 34], 3) hierarchical RNN-based model ( HRNN ) [28], 4) copy mechanism based model ( CopyNet ) [12], 5) fully-attentive model ( Transformer ) [49].
We prove the effectiveness of our Re 3 Writer by comparing the performance of the various baseline models with and without the Re 3 Writer.
Settings The model size d is set to 512.
For a patient, we directly concatenate all available patient’s health records during hospitalization as input, e.g., admission notes, nursing notes, radiology notes.
For example, if a patient only has admission notes, our model will just rely on the available admission notes to generate the PI.
We adopt Transformer [49] as the record encoder.
Based on the average performance on the validation set, the number of retrieved previous PIs, N P , is set to 20 for all three codes (see Appendix B).
During training, we use the Adam optimizer [23] with a batch size ofand a learning rate of 10 − 4 for parameter optimization.
We perform early stopping based on BLEUwith a maximum 150 epochs.
During testing, we apply beam search of size 2 and a repetition penalty [Table] [Table]
[Table]
4.2 Automatic Evaluation
Metrics We measure the performance by adopting widely-used natural language generation metrics, i [Table] e., BLEU-1, -2, -3, -4 [36], METEOR [3] and ROUGE-1, -2, -L [27], which are calculated by the evaluation toolkit [4] and measure the match between the generated instructions and reference instructions annotated by professional clinicians.
Results Table 2 shows that our Re 3 Writer can consistently boost all baselines across all metrics, with a relative improvement of 7% ∼ 20%, 4% ∼ 11%, and 5% ∼ 19% in BLEU-4, ROUGE-L, and METEOR, respectively.
The improved performance proves the validity of our approach in retrieving working experience, reasoning medical knowledge, and refining them for PI generation.
The performance gains over all of the five baseline models also indicate that our approach is less prone to the variations of model structures and hyper-parameters, proving that the generalization capabilities of our approach are robust over a wide range of models.
In Section 4.5, we verify the robustness of our approach to various data/examples.
4.3 Human Evaluation
Metrics We further conduct human evaluation to verify the effectiveness of our approach Re 3 Writer in clinical practice.
To successfully assist physicians and reduce their workloads, it is important to generate accurate patient instructions ( faithfulness , precision), such that the model does not generate instructions that “do not exist” according to doctors.
It is also necessary to provide comprehensive true instructions ( comprehensiveness , recall), i.e., the model does not leave out the important instructions.
For example, given the ground truth PI [Instruction_A, Instruction_B] written by physicians, the
PI generated by Model_1 is [Instruction_A, Instruction_B, Instruction_C], and the PI generated by [Table]
[Table] Model_2 is [Instruction_A].
Model_1 is better than Model_2 in terms of comprehensiveness , while is worse than Model_2 in terms of faithfulness .
Finally, it is unacceptable to generate repeated or otherwise unreadable instructions ( fluency ).
Therefore, we randomly select 200 samples from our
PI dataset.
The human evaluation is conducted by two junior annotators (medical students) and a senior annotator (clinician).
All three annotators have sufficient medical knowledge.
By giving the ground-truth PIs, each junior annotator was asked to independently compare the outputs of our approach and that of the baseline models, in terms of the perceived quality of the outputs - including fluency , comprehensiveness , and faithfulness of outputs compared to the corresponding ground-truth instructions.
The senior annotator re-evaluated those cases that junior annotators have difficulties deciding.
The annotators were blinded to the model that generated the output instructions.
Results We select a representative baseline, Seq2Seq, and a competitive baseline, Transformer, to show the human evaluation results (Table 3).
As may be seen from the results, our Re 3 Writer is better than baselines with improved performance in terms of the three aspects, i.e., fluency , comprehensiveness and faithfulness .
The human evaluation results show that the instructions generated by our approach are of higher clinical quality than the competitive baselines, which proves the advantage of our approach in clinical practice.
In particular, by using our Re 3 Writer, the winning chances increased by a maximum of 43 .
5 − 25 .
5 = 18 points and 55 −
= 34 points in terms of the faithfulness metric (precision) and comprehensiveness metric (recall), respectively.
At last, we further evaluate how many times physicians would have deemed the generated result as “helpful” vs. “unhelpful” in terms of assisting them in writing a PI.
The results are reported in Table 4.
As we can see, our approach can generate more accurate PIs than the baselines, improving the usefulness of AI systems in better assisting physicians in clinical decision-makings and reducing their workload.
4.4 Ablation Study
We conduct a quantitative analysis using the Seq2Seq and the Transformer for the purposes of evaluating the contribution of each proposed component: Retrieve, Reason, and Refine.
Effect of the ‘Retrieve’ Component
Table 5 (a,b,c) shows that the Diagnosis, Medication, and Procedure elements of the model all contribute to a boost in performance, which proves the effectiveness of our approach in retrieving similar patient instructions from the available repository of historical
PIs to aid in the generation of new patient instructions.
Among these three elements, Medication leads to the best improvements, which may be explained by the fact that the PI is more relevant to medications from historical experience [47].
By combining the three elements (d), we observe an overall improvement.
As a result, retrieving related working experience can boost the performance of baselines: 25.1 → 26.4 and 27.3 → 29.2 in BLEU-4 for Seq2Seq and Transformer, respectively. [Table] with [Table]
Writer
[Table]
Effect of the ‘Reason’ Component
Table 5 (e) shows that the Reason component can further improve the performance by learning the enriched medical knowledge.
By comparing (d) and (e), we can observe that the reasoned medical knowledge leads to similar improvements as the retrieved working experience does.
We attribute this to the fact that learning conventional and general writing styles for PIs in a deep learning-based approach is as important as incorporating accurate medical knowledge.
In this way, the retrieved patient instructions of previous patients can be treated as templates, providing a solid basis to generate accurate PIs.
Effect of the ‘Refine’ Component
As shown in Table 5 (f and Full Model), it is clear that the model with our Refine component performs better than the model without it, which directly verifies the effectiveness of our approach.
To further understand the ability of adaptively refining useful information, we summarize the average refining weight values ¯ λ 1 and ¯ λ 2 of Eq.
We can find that the average value of ¯ λ 2 = 0 .
42 is larger than ¯ λ 1 = 0 .
It indicates that the medical knowledge plays a prominent role in PI generation; it is in accordance with the results of (d) and (e).
We conclude that our method is capable of learning to efficiently refine the most related useful information to generate accurate PIs.
Overall, our three components improve the performance from different perspectives, and combining them can lead to an overall improvement.
4.5 Robustness Analysis
In this section, we evaluate the performance of our approach with more fine-grained datasets.
Specifically, we further divide MIMIC-III into three sub-datasets according to Gender, Age, and Disease. Specifically, to ensure an even distribution of the data, we divide the ages into three age-groups: Age < 55 (29.9%), 55 <
=
Age < 70 (30.5%), and Age >= 70 (39.7%). Table 6 shows the results of our approach on the three sub-datasets.
As we can see, the proposed approach can consistently boost baselines across different genders, ages, and diseases on all evaluation metrics, proving the generalization capability and the effectiveness of our method to different datasets/examples.
4.6 Qualitative Analysis
Figure 3 shows that our approach is significantly better aligned with ground truth instructions than the baseline.
For example, our approach correctly generates the key instructions (“ you were found to have an infection in your gallbladder ” and “ you also had a urinary tract infection which was treated with antibiotics ” (Blue-colored text)), and the personalised medication note ( “ start oxycodone 5 mg
[Figure]
Ground Truth: dear ms.
[name], you were admitted to the hospital with abdominal pain, nausea, vomiting and inability to eat.
you were diagnosed with a urinary tract infection, and treated with antibiotics.
your pain did not improve, and you were diagnosed with chronic cholecystitis (inflammation of your gallbladder).
your gallbladder was removed.
changes to your home medications include: for pain, you can take acetaminophen.
you can also take oxycodone 5 - 10mg every 4 - 6 hours for pain.
your pain should be improving, so you should require less medication over time.
Baseline: dear ms.
[name], it was a pleasure taking care of you at [hospital].
you were admitted to the hospital for abdominal pain and nausea / vomiting.
you were found to have an infection in your blood stream which we treated with antibiotics.
you were also started on iv antibiotics for this infection.
you will need to followup with dr. [name] as listed below.
with Re 3 Writer: dear ms.
[name], it was a pleasure taking care of you at [hospital].
you were admitted to the hospital with abdominal pain and found to have an infection in your gallbladder.
you were treated with antibiotics for this infection.
you also had a urinary tract infection which was treated with antibiotics.
we made the following changes to your medications : - start oxycodone 5 mg every 6 hours as needed for pain.
[Figure]
[Figure]
every 6 hours as needed for pain ” ( colored )), for the patient, who was admitted to the hospital with abdominal pain .
The baseline can only generate the correct reason for the patient’s admission, however, it also generates a serious wrong instruction (Red-colored text).
It proves the effectiveness of our approach in retrieving associated working experience and reasoning associated medical knowledge to aid PI generation.
Meanwhile, we note that our method produces interpretable refining weight values that may help understand the contribution of working experience and medical knowledge towards PI generation: when generating words similar to those that appear in the retrieved working experience (Blue-colored text) and reasoned medical knowledge (Green-colored text), the corresponding λ 1 and the λ 2 would be significantly larger than their means, ¯ λ 1 and ¯ λ 2 , to efficiently refine the relevant information from working experience and medical knowledge, respectively.
Conclusion
We propose a new task of Patient Instruction (PI) generation which attempts to generate accurate and faithful PIs.
To address this task, we present an effective approach Re 3 Writer: Retrieve, Reason, and
Refine, which imitates the working patterns of physicians.
The experiments on our built benchmark clinical dataset verify the effectiveness and the generalization capabilities of our approach.
In particular, our approach not only consistently boosts the performance across all metrics for a wide range of baseline models with substantially different model structures, but also generates meaningful and desirable PIs regarded by clinicians.
It shows that our approach has the potential to assist physicians and reduce their workload.
In the future, it can be interesting to generate personalized PIs by taking into account the patient’s cognitive status, health literacy, and other barriers to self-care.
Limitations and Societal Impacts:
The training of our approach relies on a large volume of existing PIs.
The current model performance may be limited by the size of the built dataset.
This might be alleviated in the future by using techniques such as knowledge distillations from publiclyavailable pre-trained models, e.g., ClinicalBERT [17].
Although our approach has the potential of alleviating the heavy workload of physicians, it is possible that some physicians directly give the generated PI to the patients or guardians without quality check.
Also for less experienced physicians, they may not be able to correct the errors in the machine-generated PI.
To best assist the physicians via our approach, it is required to add process control to avoid unintended usage. Acknowledgments
This work was supported in part by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre; an InnoHK Project at the Hong Kong Centre for Cerebro-cardiovascular Health Engineering; and the Pandemic Sciences Institute, University of Oxford, Oxford, UK.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work? [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have read the ethics review guidelines and ensured that our paper conforms to them.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code is available at https://github.com/AI-in-Hospitals/Patient-Instructions (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We conducted 5 runs with different seeds for all experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] PhysioNet Credentialed Health Data License 1.5.0 ( https://physionet.org/content/mimic-cxr/view-license/ 2.0.0/ ) is used.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] The code is available at https://github.com/AI-in-Hospitals/Patient-Instructions (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] All protected health information was deidentified.
De-identification was performed in compliance with Health Insurance Portability and Accountability Act (HIPAA) standards in order to facilitate public access to the datasets.
Deletion of protected health information (PHI) from structured data sources (e.g., database fields that provide patient name or date of birth) was straightforward.
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
In our work, we construct an off-the-shelf medical knowledge graph G = ( V, E ) ( V = { v i } i =1:
N KG ∈ R N KG × d is a set of nodes and E = { e i,j } i,j =1: N KG is a set of edges), which models the domain-specific knowledge structure, to explore the medical knowledge. In implementation, we consider all clinical codes (including diagnose codes, medication codes, and procedure codes) during hospitalization as nodes, i.e., each clinical code corresponds to a node in the graph. The edge weights are calculated by the normalized co-occurrence of different nodes computed from training corpus.
Figure 4 gives an illustration of the constructed medical knowledge graph.
It is worth noting that more complex graph structures could be constructed by using more large-scale external medical textbooks.
Therefore, our approach is not limited to the currently constructed graph and could provide a good basis for the future research of Patient Instruction generation.
For the constructed knowledge graph, we use randomly initialized embeddings H (0)
= { v 1 , v 2 , . .
, v N KG } ∈ R N KG ×
d to represent all node features.
To obtain the final Prior Medical
Knowledge G Pr = { v ′ 1 , v ′ 2 , . .
. , v ′ N KG } ∈ R N KG × d , we adopt graph convolution layers [35, 26, 24] to
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
encode the graph G =
( V, E ) , which is defined as follows:
[Equation]
where ReLU denotes the ReLU activation function, ˆ A
=
A
+ I is the adjacency matrix A ∈ R N KG
× N KG of the graph G with added self-connections, I ∈ R N KG × N KG is the identity matrix, ˆ D ∈ R N KG × N KG is the out-degree matrix where D ii =
(cid:80)
j A ij , W ( l ) ∈ R d × d and b ( l ) ∈
R
d are trainable parameters, and L is the number of layers.
We empirically set L = 1 and regard H (1) =
{ v ′ 1 , v ′ 2 , . . .
, v ′ N KG } ∈
R N KG ×
d as the medical knowledge G Pr ∈ R N KG × d in our Re 3 Writer.
B Effect of the Number of Retrieved Instructions
Table 8 shows that all variants with different number of retrieved instructions N P can consistently outperform the baseline model, which proves the effectiveness of our approach in exploring the prior working experience to boost the Patient Instruction generation.
In particular, when the number of retrieved instructions N P is 20, the model gets the highest performance, explaining the reason why the value of N P is set to 20 in our Re 3 Writer.
For other variants, we speculate that when N P is set to small values, the model will suffer from the inadequacy of information.
When N P is set to large values, retrieving more patient instructions will bring more irrelevant noise to the model, impairing the performance.
C Retrieve with Age and Gender Information
We further incorporate the demographic/personal information, e.g. age and gender, into our approach to match patients in the Retrieve module.
Specifically, to ensure an even distribution of the data, we divide the ages into three age-groups: Age < 55 (29.9%), 55 <
=
Age < 70 (30.5%), and Age >=(39.7%).
As a result, given a new male/female patient at 61 years old, we will match male/female patients in the age-group 55 <
=
Age < 70 in the training data to generate the PIs.
The results are reported in Table 9.
The results show that the incorporation of demographic/personal information can indeed further boost the performance, which further prove our arguments and the effectiveness of our approach.
[Table]
[Table]
D Multi-Head Attention and Feed-Forward Network
Transformer [49] including a Multi-Head Attention (MHA) and a Feed-Forward Network (FFN) have achieved several state-of-the-art results on natural language generation.
The MHA consists of n parallel heads and each head is defined as a scaled dot-product attention:
[Equation]
where X ∈ R l x × d and Y ∈ R l y × d represent the Query matrix and the Key/Value matrix, respectively;
W Q i , W K
i , W V i ∈ R d × d n and W O ∈ R
d × d are training parameters, where d n = d/n ; [ · ; · ] denotes concatenation operation.
Following the MHA is the FFN, defined as follows:
[Equation]
where max(0 , ∗ ) represents the ReLU activation function; W f ∈ R d × 4 d and W ff ∈ R 4 d × d stand for learnable matrices; b f and b ff denote the bias terms.
It is worth noting that both MHA and FFN are followed by an operation sequence of dropout [45], skip connection [14], and layer normalization [1].