M ed F ilter: I mproving E xtraction of T ask-relevant U tterances through I ntegration of D iscourse S tructure and O ntological K nowledge (2020)

Abstract: Information extraction from conversational data is particularly challenging because the task-centric nature of conversation allows for effective communication of implicit information by humans, but is challenging for machines. The challenges may differ between utterances depending on the role of the speaker within the conversation, especially when relevant expertise is distributed asymmetrically across roles. Further, the challenges may also increase over the conversation as more shared context is built up through information communicated implicitly earlier in the dialogue. In this paper, we propose the novel modeling approach MedFilter, which addresses these insights in order to increase performance at identifying and categorizing task-relevant utterances, and in so doing, positively impacts performance at a downstream information extraction task. We evaluate this approach on a corpus of nearly 7,000 doctor-patient conversations where MedFilter is used to identify medically relevant contributions to the discussion (achieving a 10% improvement over SOTA baselines in terms of area under the PR curve). Identifying task-relevant utterances benefits downstream medical processing, achieving improvements of 15%, 105%, and 23% respectively for the extraction of symptoms, medications, and complaints.

## introduction in this paper, we propose a novel modeling approach that embodies insights regarding the organization of task-oriented conversations in order to improve performance at utterance classification over sota baseline approaches. task-oriented conversations involve sharing task-relevant information that may be useful as the task ensues @xcite . unfortunately, human-to-human conversations are less well structured than expository text, which is more often the source material for information extraction and summarization. expository text is typically structured top-down and organized around information flow. task-oriented conversations, on the other hand, are typically organized around the task and knowledge of task structure provides an implicit scaffold for understanding. thus speakers feel free to elide or imply important information rather than making it explicit. the challenges have been well documented @xcite . prior work in utterance classification is a source of sota modeling approaches that perform relatively well despite these challenges while leaving much room for improvement. our evaluation in this paper specifically focuses on doctor-patient interactions. doctor-patient interactions are task-oriented, expert-layperson interactions in which the concerns voiced by the layperson (e.g., symptoms), the underlying issue identified by the expert (e.g., complaint) and the prescribed solutions (e.g., medications) play a crucial part. customer-service chats are another example of such dialogue. as in the general case, topic switching abounds: the doctor may jump from a question about a symptom to a statement providing initial assessment then back again, with or without waiting for a reply from the patient (which may, itself, be responsive or introduce a new concern). in addition, the participants make unequal contributions to different parts of the schema due to the inherent asymmetry between their roles in terms of knowledge and authority. despite these challenges, humans are able to communicate very effectively in this way. because of that, the issues increase as the conversation progresses and more shared context is built up, in part because of a certain amount of shared domain knowledge, despite differences in the extent and phrasing of it. in response to these insights, our proposed model, which we refer to as medfilter, integrates elements of dis-figure 1 : overview of medfilter. medfilter first encodes each utterance of the given conversation using a bert-based encoder (a). the obtained utterance embedding is concatenated with contextual information like speaker role, position of utterance in the conversation, and ontological knowledge (b). this is then fed to a ms-bilstm (c1) for medical relevance identification. ms-bilstm leverages speaker role information to learn speaker-specific context for each utterance. this contextual representation is concatenated with the utterance embedding (c2) and passed through another ms-bilstm (c3) which focuses on fine-grained categorization. both tasks are jointly learned. refer to section 3 for more details. our proposed method, medfilter 1 , is illustrated in figure 1 and described in detail in section 3. its architecture specifically reflects an awareness of the challenges above and begins to address them. in particular, the speaker's role (i.e., doctor, patient, and other) and position within the interaction are both introduced as structuring variables. insights from ontological knowledge are also made available through a domain ontology: specifically, the unified medical language system (umls) @xcite . from a more technical perspective, the architecture introduces a novel multi-speaker bilstm to learn role-specific context representations. medfilter also benefits from the incorporation of a hierarchical loss that jointly learns the coarse-grained task of predicting medical relevance to improve fine-grained topicbased utterance classification. the ability to extract medically relevant utterances from doctor-patient conversations and categorize them into the medical topics/categories has a substantial practical impact in medical practice @xcite . 2 related work dialogue summarization: in addition to the challenges noted earlier in the paper, other linguistic phenomena such as backchannels, false starts, and topic diffusion are prominent in human-tohuman conversations. they add noise, which challenges the capabilities of otherwise effective sumarization approaches such as pointer-generator networks @xcite . some prior work has relied on an information extraction (ie) based approach to extract details about individual medical entities such as symptoms or medications @xcite . however, recently, multiple studies @xcite @xcite have shown the benefits of using the topical structure in goal-oriented dialogues to improve summarization. within that scope, @xcite introduce keypoint sequences that describe the logical topic flow of the summary of customer-service chats. they propose a hierarchical transformer to predict these topics (key-points) for each utterance and use them as auxiliary labels to guide the summarization. this past work inspires our work in which we extend the approach and then apply it in the more challenging domain of doctor-patient interactions. we consider it more challenging both in terms of the number of utterances per conversation (avg. 225 vs 20) and topic switches @xcite . to improve the key-point sequence utterance-level topic classification approach @xcite , we propose medfilter that models speaker-specific context augmented with ontological knowledge and a hierarchical loss function. intent classification: the problem of classifying utterances into medical topics/categories has many similarities with the task of utterance-level intent classification @xcite @xcite . in our case, medical categories act as coarse-grained intents that drive the content of the discussion. much of the previous work in intent classification caters to creating better dialog agents that condition their responses on the intent of the previous utterance @xcite . for instance, @xcite ; @xcite propose intent classification as a text classification task where each utterance is considered a complete, independent command. however, this is not true in our case as the discussion about a medical category might range over multiple utterances, each dependent on context. hence, we tackle the classification problem as a sequence-labeling task. sequence labeling in dialogue: most prior work that employs sequence labeling for utterance classification in dialogues @xcite @xcite evaluates their systems on dialogue-act classification @xcite or emotion recognition datasets @xcite . in this paper, we adopt stateof-the-art modeling approaches from the emotion recognition task @xcite to serve as baselines in our evaluation since our task has not previously been benchmarked. ## proposed method: medfilter the overall architecture of medfilter is shown in figure 1 . the input to medfilter is a transcribed clinical conversation c of form {u 1 , u 2 , ..., u n }, where each u i represents an utterance. each utterance in the conversation is passed through a bert-based encoder (fig. 1a and sec. 3.1) to get a fixed-dimensional representation. contextual information such as speaker role, the utterance's position in the conversation, and ontological knowledge (fig. 1b and sec. 3.2) is then appended to the bert representation. the encoding is input to the coarse multi-speaker bilstm (ms-bilstm) model (fig. 1c1 ) followed by a fully-connected layer to classify the relevance of utterances for topical classification. the representation created by ms-bisltm (coarse) is then concatenated with the utterance encoding (fig. 1c2 ) and the resulting vector is fed to the fine-grained ms-bilstm (fig. 1c3 ) to classify utterances into different medical categories (sec. 3.3). medfilter is jointly optimized on both classification tasks. ## impact of utterance classification on downstream medical extraction the results in the previous section portray the effectiveness of medfilter at sorting important utterances in clinical conversations into medically relevant categories. such filtering, when included in the pipeline (for example, as a pre-processing step), can assist downstream medical processing methods to focus on utterances that contain information pertinent for their tasks (figure 2 ), by improving the signal-to-noise ratio in the input. in this section, we evaluate whether the use of medfilter to prune irrelevant utterances is advantageous for symptom, medication, and complaint extraction. ## discussion why does contextual information help? ablation results (figure 3 ) show that incorporating speaker role information and umls semantic-type information provides significant improvements in auc scores for utterance classification. in figure 4a , we plot the proportion of utterances from different medical categories against their speakers. while both parties contribute equally to symptom discussions, there is a clear asymmetry in the number of medication and complaint utterances spoken by the doctor and the patient, explaining the contribution of speaker role information in differentiating medication/complaint utterances from others. we also plot the distribution of the four most frequent umls semantic types present in the utterances of different medical categories (figure 4b ). for medications, we find that umls entities with semantic type pharmacologic substance are present in more than 55% of the medication utterances indicating that its detection is a knowledge-dependent task. similarly, and supporting our hypothesis, disease/syndrome and sign/symptom are the most frequent semantic types in complaint and symptom utterances, respectively. ## future work for a system to correctly classify the samples of the above two categories, it needs both to generalize to patient-generated language, and to have a semantic understanding of whether the description strays from normal. incorporating data from online self-disclosure sites like medical subreddits and discussion forums @xcite during training might prove beneficial for learning better representations for such vocabulary. concept normalization data sets @xcite could also be leveraged in this regard. our approach of training the bert encoder separately from the context encoder would allow medfilter to learn from such non-dialogue resources. extraction tasks (section 6) mostly evaluate the ability of medfilter to recognize utterances that contain the most information about the name or type of the medication, symptom, or complaint. however, to quantify the context-level benefits of medfilter, especially the speaker-specific context modeling (ms-bilstm), on downstream processing, we need to evaluate the system on problems like regimen extraction (selvaraj and konam, 2019) or symptom summarization @xcite . such tasks require utterance classification models to correctly identify utterances that discuss fine-grained details about the topic and would therefore evaluate a model's ability to solve multiple challenges like coreference resolution, speaker-specific context detection, thread identification, etc. such an evaluation is a part of future work. ## conclusion in this paper, we have proposed a novel text classification approach that specifically leverages insights into the organization of task-oriented conversations in order to improve performance at topicbased utterance classification over sota baseline approaches. in particular, we have demonstrated that our utterance classification model, medfil-ter, benefits from discourse information, domain knowledge, speaker-specific context modeling, and a hierarchical loss to reach a new state-of-theart performance on a doctor-patient interactions dataset. we find that using topic-based utterance classification in general, and medfilter in particular, as a pre-processing step before medical extraction tasks, significantly improves the extraction scores. we believe that the contributions made in this work would also generalize to other kinds of expert-lay dialogue like customer-service chats. the de-identified doctor-patient dialogue corpus used in this work was made available by university of pittsburgh medical center (upmc) and abridge ai inc.. most of the conversations in this corpus are follow-up encounters between cardiovascular/general medicine doctors and patients. figure a1 shows the distribution of the number of utterances in each conversation. the number ranges from as low as 3 to as high as 1521 with a mean of 225. the proportion of medically relevant utterances in a conversation is quite low (table a1 ). as shown, utterances that belong to the three categories combined make up less than 10% of the conversation portraying the amount of noise present in doctor-patient conversations with regards to further medical processing. category #mr-utt #mr-utt/#utt (%) complaints 6.17 (3.40) 4.34 (6.06) symptoms 3.56 (4.00) 1.98 (2.37) medications 4.79 (3.70) 3.10 (5.49) table a1: avg. (std. dev.) medically relevant utterances (mr-utt) in each medically relevant category. in table a2 , we show the average position in the doctor-patient conversation where the speakers start discussing different medical topics. several of the the encounters are follow-up discussions about a pre-existing complaint. therefore, patient's current condition with respect to the complaint is often discussed earlier in the conversation. this is generally followed by a discussion about different body systems and associated symptoms that may be bothering the patient, which allows the doctor to prescribe suitable medications. category relative position complaints 0.133 (0.043) symptoms 0.321 (0.057) medications 0.524 (0.069) table a2: avg. (std. dev.) relative position in the conversation where speakers start discussing different medical topics. the above-mentioned flow is merely an ideal depiction of the logical path that could be followed in the dialogue. however, real conversations in the corpus contain multiple topic-switches. for example, discussion of a symptom could be followed by medication which could then lead into a discourse about another symptom and so on. utterance labels 1 check if conversation can be added 1 dr: good morning. 2 pt: good morning. 3 dr: i'm here with, [patient name]. 4 dr: last time i saw you, you were getting pains in your left leg. is it still the case? c,s 5 pt: yes, i do. s 6 dr: okay, and generally, what are you doing when you get the pains? 7 pt: um, usually just a heating pad or, you know, ice. 8 dr: right, but what causes the pains, is what i was getting at? s 9 pt: uh, i think just the strain of, like, walking, or, or exercise. s 10 dr: all right. 11 dr: i think i am going to ask you to try some baclofen. m 12 dr: this is a patch you put on the foot when it's bothering you. m 13 dr: try one patch. m 14 dr: it'll last up to 6 hours. 15 pt: okay. 16 dr: if you like it, let me know. 17 dr: we'll get you a prescription. 18 pt: okay. 19 dr: the difference is, you can put this exactly where you need it on the foot and since it's going through the skin, it's not rough on your stomach like, let's say, ibuprofen or aspirin or any of the over the counter stuff would be. 20 pt: okay. table a3 : a constructed example conversation (@xmath0, @xmath1, and @xmath2. because conversations in the corpus cannot be published or distributed without agreement, the example here is based on a corpus conversation but with the details changed.