Large Scale Sequence-to-Sequence Models for Clinical Note Generation from Patient-Doctor Conversations (2023)

Abstract: We present our work on building large scale sequence-to-sequence models for generating clinical note from patient-doctor conversation. This is formulated as an abstractive summarization task for which we use encoder-decoder transformer model with pointer-generator. We discuss various modeling enhancements to this baseline model which include using subword and multiword tokenization scheme, prefixing the targets with a chain-of-clinical-facts, and training with contrastive loss that is defined over various candidate summaries. We also use flash attention during training and query chunked attention during inference to be able to process long input and output sequences and to improve computational efficiency. Experiments are conducted on a dataset containing about 900K encounters from around 1800 healthcare providers covering 27 specialties. The results are broken down into primary care and non-primary care specialties. Consistent accuracy improvements are observed across both of these categories.

## introduction medical documentation plays an important role in diagnosis, treatment, and delivery of safe patient care. healthcare professionals are required by law to document their encounter with the patient. apart from this, medical documentation is also useful in research and driving quality improvement @xcite . medical documentation needs to be accurate and comprehensive, capturing the patient history, physical examination, laboratory and imaging studies, diagnosis, and treatment options. physicians typically spend 35% of their time on documenting the patients visits of the day @xcite . this increased documentation burden is one of the main causes for physician burnout @xcite . the use of automatic speech recognition (asr) systems have simplified document creation to a great extent where physicians dictate medical notes into electronic health records (ehrs). the content in the dictation is by and large already discussed with the patient albeit in colloquial language. advances in deep learning in the field of natural language processing has attracted increased attention in generating medical reports directly from patientdoctor conversations @xcite @xcite @xcite . some of the challenges posed by this problem are: encouraged by ongoing advancements in neural sequence transduction (e.g., for machine translation and abstractive summarization), we follow an end-to-end approach to the problem. we use a single transformer model to generate clinical reports directly from patient-doctor conversational transcripts with various enhancements to handle long input and output sequences. our approach is similar to @xcite where a transformer model with pointer generator was used to generate clinical notes for orthopedics. we extend this with ## dataset we use a dataset consisting of medical encounters across 27 medical specialties in the ambulatory setting. each encounter includes a patient-doctor conversation transcribed and diarized by an automatic speech recognition (asr) system. the asr transcript is used to generate three sections of a medical note, namely history of present illness (hpi), assessment and plan (ap), and physical examination (pe). the median number of words in each of these sections is 166, 291, and 111 respectively; while that for the transcript is 2128. the dataset is collected across 128 medical institutions and 1811 physicians. ## modeling we use a sequence-to-sequence model with transformer architecture @xcite and train a separate model for each of the three sections. since the report format and style varies across specialties and physicians, each transcript is prepended with a unique specialty id and doctor id to condition the report generation. in all of our experiments we use the big model size, similar to the one specified in @xcite with 16 attention heads in each multi-head attention module, inner representations of size 1024, and the feed-forward layer size of 4096 in each transformer layer. we, however, use an 8 encoder layers and 4 decoder layers configuration instead of the default 6-6 one since the transcripts are longer and have a higher perplexity language than the reports. we use prelayer normalization @xcite and the pointing mechanism @xcite . for positional encoding, on encoder side we use rotary positional embeddings (rope) @xcite and on decoder side we use the t5 scalar relative positional embeddings @xcite . we make several changes over this baseline model in order to further tailor it to our problem as discussed in the following subsections. ## evaluation metrics we report three f1 score-based accuracy metrics: (a) rouge-l: this is our implementation of the rouge metric @xcite in which we check for the longest common subsequence between the reference and hypothesis; (b) fact-c: this measures the overlap of core medical facts, e.g., pain, automatically extracted from the hypothesis and reference; (c) fact-f: this reflects the match of full medical facts, including attributes, e.g., laterality, body part. ## experiments we trained our models on 4 × 80gb gpu machines with data-parallel training using the fairseq library @xcite . each model was trained for a predefined number of steps on the train partition, and decoded and scored at multiple checkpoint intervals. the test partition contains the chronologically latest encounters for each physician, while dev contains the set of encounters just before test for each physician. we also create a smaller subset of the train partition called recent that consists of the latest 200 encounters for each physician. it is used to fine-tune the trained model to the most recent encounters so as to bias it to1 . specialties with fewer encounters were sampled more often during training. we averaged the last 10 model checkpoint weights to reduce the variance in results and picked the best performing averaged checkpoint on the dev set to report test results. for all experiments we use a vocabulary size of 45k tokens which is shared for the source and the target. encoder and decoder token embeddings are also shared. we calculate and report micro averages which are broken down into 1) primary care specialities, which consist of family medicine and internal medicine, and 2) all other specialties that we refer to as non-primary care. the primary care specialties deal with a broad set of diseases and conditions for people of all ages and are thus harder to model. for tokenization experiments, we trained the sentencepiece model on the train partition. token length was restricted to 100 characters. we also reserved certain words to be included in the vocabulary, such as the specialty ids, patient ids and speaker turn indicators. for chain-of-clinical-facts experiments, we prepended the facts to the summary with @xmath0, with individual fact phrases separated by @xmath1. on an average, the length of the prefix is about 20% that of reports, excluding the separator tokens. for the contrastive loss training process, we generated 20 candidate summaries for each encounter in the recent partition using the base model that was trained with cross-entropy loss. we applied nucleus sampling with a probability mass of 0.6 to generate these summaries. we then ranked the summaries based on their average rouge-l and fact-c scores, with the highest-scoring summary being ranked first. finally, we fine-tuned the base model using an equal-weighted combination of the con-trastive and cross-entropy loss. during fine-tuning, we dynamically chose 8 out of the 20 candidates for each example in the batch for computation and memory efficiency, where the top ranked hypothesis was always kept, while the rest 7 were sampled randomly. ## results the accuracy for each of the three sections by incrementally adding the various modeling technique is shown in the use of subword and multiword tokenization, apart from improving accuracy, also helps to speed up model convergence as seen in with the use of flash attention, we were able to increase the number of tokens per batch by 4x yielding a training speed-up of 2-2.5x times in terms of number of tokens processed per second. during inference, query chunked attention enables processing transcripts of any length without truncation as opposed to vanilla attention which runs out of memory on a 16g gpu for inputs longer than 10k tokens. ## conclusions we used transformer-based models to build a largescale, multi-specialty, end-to-end abstractive summarization system capable of generating medical reports from conversations. we presented various modeling and efficiency improvements that can be applied to better adapt these models to this challenging task.