Towards Automating Medical Scribing : Clinic Visit D ialogue2 N ote Sentence Alignment and Snippet Summarization (2021)

Abstract: Medical conversations from patient visits are routinely summarized into clinical notes for documentation of clinical care. The automatic creation of clinical note is particularly challenging given that it requires summarization over spoken language and multiple speaker turns; as well, clinical notes include highly technical semi-structured text. In this paper, we describe our corpus creation method and baseline systems for two NLP tasks, clinical dialogue2note sentence alignment and clinical dialogue2note snippet summarization. These two systems, as well as other models created from such a corpus, may be incorporated as parts of an overall end-to-end clinical note generation system.

## introduction as a side effect of widespread electronic medical record adoption spurred by the hitech act, clinicians have been burdened with increased documentation demands @xcite . thus for each visit with a patient, clinicians are required to input order entries and referrals; most importantly, they are charged with the creation of a clinical note. a clinical note summarizes the discussions and plans of a medical visit and ultimately serves as a clinical communication device, as well as a record used for billing and legal purposes. to combat physician burnout, some practices employ medical scribes to assist in documentation tasks. however, hiring such assistants to audit visits and to collaborate with medical staff for electronic medical record documentation completion is costly; thus there is great interest in creating technology to automatically generate clinical notes based on clinic visit conversations. not only does the task of clinical note creation from medical conversation dialogue include summarizing information over multiple speakers, often the clinical note document is created with clinicianprovided templates; clinical notes are also often note dialogue she declines the [qa-1] doctor: have you had a pneumonia vaccine? pneumonia vaccine. [qa-1] patient: no, i don't think so. ## background ## corpus creation data the data set was constructed from clinical encounter visits from 500 visits and 13 providers. the data for each visit consisted of a visit audio and clinical note. for each visit audio, speaker roles (e.g. clinician patient) were segmented and labeled. automatically generated speech to text for each audio was manually corrected by annotators. we also combined our data with external data, the mock patient visit (mpv) dataset, from @xcite annotations each annotation is based on a clinical note sentence association with multiple transcript sentences. a note sentence can be associated with zero transcript sentences and an inferred-outside label for default template values, e.g. "cv: normal". one may also be associated with sets of transcript sentences and a set tag, e.g. dictation or qa (described below). finally, when multiple sets have anaphoric references, they may be tied together using a group label. given this hierarchy, the annotation related to a single note sentence can be represented as a tree as shown in figure 1 . set labels command: spoken by the clinician to the scribe to make a change to the clinical note structure, e.g. if after applying all possible associations and still there is information in the note sentence not available from the transcript, then an incomplete tag is added. a note sentence is left unmarked if no information can be found from the transcript. to measure interannotaor agreement, we calculated the triple, path, and span metrics introduced in @xcite , briefly described again here. the triple, path, and span metrics were defined based on instances constructed from the annotation tree representation. specifically, for the triple metric, which measures unlabeled note to dialogue sentence match, instances are defined by note sentence id and transcript sentence id per visit, e.g. 'visitid_01|note_0|3'. the second metric, similar to the leaf-ancestor metric used in parsing, takes into account the full path from one note sentence to one dialogue sentence, e.g. 'visitid_01|note_0|group|qa|3'. the span metric, similar to that of parseval, measures a node-level labeled span of dialogue sentences, e.g. for the top group node would be 'visitid_01|note_0|group| @xcite @xcite ' @xcite . when testing agreement, labels for each annotator are decomposed to these instance collections; true positive, false positive, and false negatives may be counted by the matches and mismatches between annotators. f1 score is calculated as usual. the different definitions allow both relaxed (triple) and stricter (path and span) agreement measurements. labeling process a group of 11 annotators were trained for various parts of the processing task. audio transcription was performed using elan (archive.mpi.nl/tla/elan) and dialogue2note annotation was performed using an in-house software. annotators underwent training on sample files for which they received in-depth feedback. they additionally took a training quiz and self-reviewed errors. after training, their interannotator agreement was calculated based on 10 final files. their average pairwise triple, path, and span f1 scores were 0.754, 0.549, and 0.645 respectively, a reasonable performance given the task difficulty. 3 annotation statistics on average 58 ± 18 % of the clinical note was marked with an annotation. this suggests that around 40% of the note is structural, e.g. blank sentences or section headers, or from outside sources, e.g. injected labs, medication lists, etc. on average 13 ± 12 % of the transcript sentences were marked. this low number suggests that much of the information from transcripts consisted of repeats or were unused. to quantify this, we calculate the percentages of annotations which annotates cross one, three, or five other annotations 4 ( the full amount of annotations from the dia-logue2note labels may be used to create classifiers in many different types of tasks, e.g. dialogue relevance classification, topic segmentation, command identification, etc. however, in the remaining sections, we focus on two particular system applications : automatic dialogue2note sentence alignment and snippet summarization. for these baselines, the train and test sets were split using stratified random sampling using an 80-20 split. the training and test sets were composed of 400 and 100 of our visits; 53 and 13 for the mpv visits. 91 visits from training was reserved for development testing. as a simplification, the group, incomplete, and command labels are ignored for these baselines. ## sentence alignment baselines we define the dialogue2note sentence alignment baseline task as the classification of 1-to-1 dialogue sentence and clinical note sentence pairs with set labels. thus, the candidate space includes all combinations of clinical note sentences paired with all dialogue possible sentences in a visit; only those annotated with labeled associations are considered positive. this is a subset of the full annotation tasks that require 1-to-many multi-label classifications with hierarchical group set labels. however, this feature description match-note dot product of note and transcript vector divided by the magnitude of the note vector. match-transcript dot product of note and transcript vector divided by the magnitude of the transcript vector. bitext corpus creation related work the topic of bitext corpus creation is often used in the context of creating resources for statistical machine translation or as a means to create cross lingual linguistic resources @xcite ; it is also used to describe simplification dataset creation @xcite @xcite . while highly parallel bitext can be aligned using sentence length methods, much like other comparable corpora alignment strategies, multi-form comparable corpora cannot rely on monotonic ordering or correlated bitext sentence length; moreover the different text forms presents additional constraints on exact narrative structure. like in previous work, we build our baselines for dialogue2note sentence alignment by using similarity features with some adjustment to incorporate similarity over multiple sentences. system description candidate classification instances for every note sentence and transcript sentences pair were created and classified into one of the previously described set labels. for each clinical note, an additional classification instance was created for a match with an empty transcript line. (this occurs with a inferred-outside label). a single tag was assigned to each classification instance according to annotated labels. if multiple tags existed per sentence pair, we took the first label in the following order: statement, statement2scribe, qa, dictation. sentences were tokenized, changed to lemma form using spacy english model (spacy.io), and vectorized according to a bag of words model. stop words and punctuation were removed. to balance the uneven data distribution, the number of negative class instances were sampled randomly according to configurable parameter, neg_samp. we experimented with three baseline pairwise classification systems: to avoid erroneous matches to answer sentences, in this system, common answers (e.g. "no") were removed from the train set. in the current system, classifications for each note-dialogue sentence pair are labeled independently. we can improve the system by framing the required matches for each clinical note sentence as a sequence labeling problem. more semantic normalization features and surrounding sentence features would benefit the classification. finally, in the future we can try more complex sentence vector representations. ## snippet summarization baselines we define the snippet summarization baseline task where given the gold standard dialogue snippet text, a corresponding clinical note sentence is generated. the number of instances of aligned sets for train, dev, and test was 7129, 1851, and 2085 respectively. the average number of input and output tokens was 24 and 13 respectively. ## conclusions in this work, we provided baselines for two tasks that work towards natural language generation of note sentences from medical visit conversation. an automated dialogue2note sentence alignment system can be used to create realistic training data so immensely critical for modern systems. meanwhile, if given properly extracted transcript snippets, dialogue2note snippet summarization could provide a valuable building block for an overall language generation system. in future work, additional metadata information, (e.g. set labels, speaker, specialties) may be incorporated into the network architecture. although we only explore two systems here, other models such as topic segmentation, extractive summarization, note sentence ordering, and dialogue command classification, can be trained from this annotated dataset alone. these labels may alternatively be used for additional multitask classification objectives in a full sequence-to-sequence model. extension of this labeled dataset may yield further interesting gains. for example, textual entailment labels between paired snippets would allow progress towards understanding and generating semantic variations and detail. event annotation, which structures text, if performed on paired snippets, would provide training examples for data-totext or text-to-data generation. together or apart, such systems would enable automation of clinical note generation whether as a full end-to-end solution or as piecemeal suggestions in a human-augmented solution. ultimately this technology may be utilized to deburden clinicians, allowing them to focus back on patient care.