Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue (2023)

Abstract: Factual correctness is often the limiting factor in practical applications of natural language generation in high-stakes domains such as healthcare. An essential requirement for maintaining factuality is the ability to deal with rare tokens. This paper focuses on rare tokens that appear in both the source and the reference sequences, and which, when missed during generation, decrease the factual correctness of the output text. For high-stake domains that are also knowledge-rich, we show how to use knowledge to (a) identify which rare tokens that appear in both source and reference are important and (b) uplift their conditional probability. We introduce the “utilization rate” that encodes knowledge and serves as a regularizer by maximizing the marginal probability of selected tokens. We present a study in a knowledge-rich domain of healthcare, where we tackle the problem of generating after-visit care instructions based on patient-doctor dialogues. We verify that, in our dataset, specific medical concepts with high utilization rates are underestimated by conventionally trained sequence-to-sequence models. We observe that correcting this with our approach to knowledge injection reduces the uncertainty of the model as well as improves factuality and coherence without negatively impacting fluency.

## introduction recent advances in language modeling (c.f. @xcite ; @xcite for survey) have enabled applications across multiple domains including education @xcite , jurisprudence @xcite , e-commerce @xcite , and healthcare @xcite @xcite . one of the central challenges in deploying these models in-the-wild is that rare words tend to have underestimated conditional probability during generation @xcite @xcite . however, in high-stakes applications, many of these rare words are semantically important and need to be preserved. for example, some symptoms, diseases, and medications can be both rare and important @xcite ) (e.g. knowing that the patient is taking warfarin is extremely important, even if the word "warfarin" occurs infrequently). prior approaches for handling rare word generation utilize a copy mechanism @xcite @xcite . this facilitates copying from the source text using a probabilistic switch to decide if the next output token is generated or copied from the input @xcite . however, it doesn't properly resolve the main challenge: not all rare tokens are important. only specific rare tokens (e.g. warfarin) have a high probability of appearing in the reference sequence when found in the source sequence. in cases where the training data does not have enough structure to disambiguate which rare words are essential, the copy mechanism becomes overly extractive @xcite . also relevant to this paper are previous works that integrate knowledge into language models @xcite . in entity-centric summarization, @xcite ; @xcite add key phrases to the prompt, which through the self-attention mechanism influence the output distribution. however, for prompts containing rare tokens, self-attention struggles to capture the prompt-reference dependency, and the marginal probability of rare tokens remains underestimated. @xcite extends this approach by not only explicitly including the medical concepts in the input sequence, but also adding a related term to the loss function. however, they still find that for rare tokens the model underestimates the conditional probability during generation. finally, dictionary look-up of rare and out-ofvocabulary words has been studied in @xcite ; @xcite . however, these papers focus on finding good representations of specific tokens. in this paper, we tackle the problem of uplifting important rare tokens even when a good representation is not available. we base our work on the premise that specific rare tokens (e.g. warfarin) have a high probability of appearing in the reference sequence if they also appear in the source sequence. the main questions we tackle in this paper are the following: how do we know which rare tokens have a propensity to appear in both the source and the reference? how do we encode this information into the model? we study our approach in the healthcare setting, for the concrete problem of after-visit care instruction generation from a medical dialog between patient and medical professional. we define the medical concept utilization rate and utilization-rate-aware training objective in section 2, discuss the care plan generation problem and data collection in section 3, describe the sequence-tosequence model setup in figure 4 , and report experimental results in section 5. our contributions are the following: we observe performance improvement with both automatic metrics and human evaluation with medical experts. ## approach in many sequence-to-sequence tasks, certain rare concepts have a high probability to appear in the reference sequence (y) if they also appear in the source sequence (x). we call these concepts "high utilization concepts" (@xmath0. these concepts are comprised of one or more tokens @xmath1, ν 1 , ...]. we hypothesize that a source of factuality errors in many sequence-to-sequence tasks is that learned model underestimate the conditional probability of high utilization concepts p(y @xmath2, |@xmath3, x, ν ∈ c, @xmath4, @xmath5...), where p denotes the model estimated probability and p is the true probability. definition 2.1 (high utilization concepts) given a universe of concepts c, the set of high utilization concepts c hu is defined as equation 1 answers the question "how do we know which rare tokens have a propensity to appear in both source and target?" while at the same time it works for rare tokens. this key insight leads us to define two goals for this work: learn to identify high utilization concepts, and build a utilization-rate-aware training objective. ## after-visit care instruction generation: task and data description after-visit care instructions (care plan) are a set of actions (instructions) that a medical professional writes in the patient's electronic health record (ehr) as a followup to the patient's visit. a care plan often includes a list of medications with appropriate directions, further medical evaluations, or educational information for preventive care. before writing the care plan, the medical professional discusses it with the patient, and together, they jointly agree on the next course of action. this joint decision-making implies that most of the necessary information for writing the care plan is already available in the conversation. in figure 1 , we show two examples. in each example, we present the (a) segment of the conversational dialog corresponding to provider messages discussing the care plan with the patient and (b) corresponding care plan charted in the ehr. we can see that the instructions finding medication disorder class character thing history activity procedure laterality course qualifier demographic topic measurement laboratory onset timing surgery semantic types 0.00 0.01 0.02 0.03 0.04 0.05 marginal probability (a) average marginal probability of every semantic type. finding medication disorder class character thing history activity procedure laterality course qualifier demographic topic measurement laboratory onset timing surgery semantic types 0.0 0.2 0.4 0.6 0.8 1.0 utilization rate (b) semantic type utilization rates. are written in a directive format, using action verbs and often paraphrasings of the corresponding text in the dialogue. the care plan does not always have all the medical concepts mentioned in the conversation. in the first example, "serotonin syndrome" and "celexa" are rare, but the care plan includes only the latter. we need a model that is robust to rare medical concepts and can discern which knowledge needs to be carried forward. we tackle the problem of taking the relevant section in the conversations corresponding to the care plan as input and automatically derive care plan instructions that the medical professionals can approve. we do not assume access to 1-1 mappings between the sentences in the conversation to the care plan instructions. however, we develop a method to derive a dataset of 1-1 mappings, albeit noisy, which we use for model training. dataset construction. we use a dataset with 14k medical professional-patient encounters collected on a virtual primary care platform. each encounter has a text-based conversation between the medical professional and the patient. we applied an in-house conversation discourse parser to extract only those dialogue turns from the medical professional's corresponding to the care plan discussion. we also have the associated care plans written from the patient's electronic health record for that encounter. on average, each encounter has 9 dialogue turns corresponding to care plans and 4 care plan instructions. we need a parallel corpus with pairs of dialogue turns and care plan instructions for our model. getting manual annotations for each encounter would be expensive as it requires expert knowledge. therefore, we automatically construct a paired dataset, albeit noisily, from the paired encounter level care plan and provider dialog turns. we get sentence-level embeddings for every sentence in each turn and instructions in the care plan and pair those with the highest cosine similarity (we provide additional details in the supplementary material). at the end of this, we have 48,000 source-reference pairs, where the source is a sentence in the conversational dialog and reference is the mapped instruction. we randomly sample 3000 pairs for testing, 1000 for validation, and the remaining 44,000 pairs for training. we use medical concepts from umls @xcite and in particular snomed-ct and rxnorm ontologies. the synonyms are pooled from all ontologies in umls that map to the corresponding concept in snomed-ct and rxnorm. to identify the concepts, we use an in-house lookupbased concept recognizer. it uses a sliding window strategy to find maximal matches of text corresponding to medical concepts and their synonyms. it ignores stop words while doing the match. finally, it has an agglomeration step that leverages a concept hierarchy. if we have overlapping spans corresponding to two concepts where one is a child of another (eg "lower abdominal pain" and "abdominal pain") then only the more specific concept is extracted. if two different concepts have a span overlap and are not hierarchically related, then the concept linking is greedily selected with the concept on the left being given priority. identifying high utilization concepts. we limit c sel to only medical concepts and choose ϕ such that it maps them to their snomed ct semantic types (which informs our choice of e). in our case study this narrows down 758 unique medical concepts to their 19 semantic types. the marginal probability p(@xmath6. comparing them we can see that utilization rates are 10-100x larger than the marginal probabilities. this suggests that all medical concepts are part of high utilization tokens set (c @xmath7. it also means that many kinds of medical concepts that are present in the source sequence do not get generated in the output sequence, which drastically hurts medical correctness. ## experimental setup we follow the standard practice @xcite of training our sequence-to-sequence models using fairseq framework @xcite . we use byte-pair encoding implemented in the fastbpe package @xcite . we use a transformer architecture for our model and train models on our data from scratch foot_0 . ## conclusion in this work, we tackle the problem of under-generation of rare but important tokens in sequence-to-sequence models. we show that external knowledge can be effectively injected into the sequence-to-sequence models and mitigate the problem of lexical precision. we characterize the problem by identifying a set of lowfrequency but important concepts and defining their utilization rate, which estimates the probability of a concept that is present in the source to be also present in the reference. we confirm that modern welltrained sequence-to-sequence models suffer from underestimating utilization rates, and propose a way to directly maximize it during training. we design a differentiable proxy based on the marginal entropy and propose a regularized training objective. since some concepts may be omitted from the reference, we extend the approach by applying weights, which restrict the we perform a case study in automatic care plan generation from medical dialogues. we experiment with a custom internal dataset and observe the effectiveness of the approach. we also compare a previous approach for external knowledge injection -dynamic beam allocation (dba). first, we find that regularization improves the model's utilization rate by pushing it closer to the empirical values observed in reference sequences. second, regularization reduces the model's uncertainty at early timesteps: exactly where concepts are typically introduced. third, we observed a significant (in terms of standard deviations) quality improvement. more specifically, we did a human evaluation of relevance, concept overlap, medical usability, and fluency using five medical experts. the results revealed the enhanced relevance and usability of generated instructions while, unlike dba, maintaining high fluency and low degeneracy. ethics statement: this work was done as part of a quality improvement activity as defined in 45cfr §46.104(d)(4)(iii) -"health care operations" secondary research. reproducibility statement: code used for training regularized sequence-to-sequence models in this paper is available at https: //github.com/curai/curai-research/ tree/main/careplan-charting. however, data will not be shared due to patient privacy and hipaa compliance. as it contains significant amount of patient health information (phi) and cannot be shared. privacy concerns: our research aims to utilize knowledge to enhance nlg systems. however, we also acknowledge the privacy concerns associated with leveraging sensitive medical information. all training data was anonymized during preprocessing step, and all personally identifiable information (pii) was removed to protect patient identities in generated outputs. another privacy consideration is inference leakage, where nlg systems unintentionally reveal sensitive information during generation. we suggest incorporating differential privacy mechanisms to prevent the association of rare tokens or medical concepts with specific individuals.