Adapting Event Extractors to Medical Data: Bridging the Covariate Shift (2021)

Abstract: We tackle the task of adapting event extractors to new domains without labeled data, by aligning the marginal distributions of source and target domains. As a testbed, we create two new event extraction datasets using English texts from two medical domains: (i) clinical notes, and (ii) doctor-patient conversations. We test the efficacy of three marginal alignment techniques: (i) adversarial domain adaptation (ADA), (ii) domain adaptive fine-tuning (DAFT), and (iii) a new instance weighting technique based on language model likelihood scores (LIW). LIW and DAFT improve over a no-transfer BERT baseline on both domains, but ADA only improves on notes. Deeper analysis of performance under different types of shifts (e.g., lexical shift, semantic shift) explains some of the variations among models. Our best-performing models reach F1 scores of 70.0 and 72.9 on notes and conversations respectively, using no labeled target data.

## introduction events are an important phenomenon in the field of computational semantics. they offer an intuitive mechanism for constructing structured representations of text, which can be used for downstream tasks such as question answering and summarization. events also embody a crucial function of language: the ability to report happenings. narratives from many diverse domains (e.g., news articles, literary texts, clinical notes) use events as basic building blocks. these characteristics make event extraction a key sub-task of interest for text understanding pipelines in multiple domains. despite its importance, building high-performing and generalizable systems for event extraction has remained an elusive goal. one of the major hurdles is that the notion of what counts as an important event is usually task-specific or domain-specific (sometimes both). for example, to build a system that can track a patient's disease progression from clinical notes, event extractors only need to focus on extracting medical events relevant to that illness. this task/domain specificity has encouraged prior work to focus on specific event types @xcite @xcite or domains @xcite . owing to this narrow focus, supervised event extractors often fail to adapt to new domains or event types @xcite . unsupervised event extractors that use syntactic rulebased modules @xcite , conversely, have a tendency to over-generate by labeling most verbs and nouns as events. in this work, we try to achieve a balance between these extremes by adapting event extractors using unsupervised domain adaptation techniques. we also study the behavior of these techniques under various types of linguistic shifts (e.g., lexical shift, semantic shift) to gain insight into differences among them. exploring adaptability under no (or little) supervision is crucial, since sourcing annotated data for new domains, especially medical texts, can be expensive and time-consuming. following prior work, we formulate event extraction as the task of labeling triggers, i.e., words which instantiate an event @xcite . for example, in the sentence "she was diagnosed with cancer," diagnosed and cancer are triggers, referring to "diagnosis" and "illness" events respectively. throughout our work, we model event trigger labeling as token-level classification. to test adaptability, we create new event extraction test sets using english texts from two diverse medical domains: (i) clinical notes, and (ii) doctorpatient conversations. we develop comprehensive event annotation guidelines, based on timeml @xcite and thyme-timeml @xcite ( §3), and use them to annotate 45 documents from each domain. as a baseline, we train a bert-based event extraction model on english news articles from timebank @xcite , which is labeled using timeml, and test its performance on our datasets. to improve this out-of-domain baseline performance, we tackle the problem of covariate shift, i.e., differences between marginal distributions of source (news) and target domains (notes or conversations). we experiment with three marginal alignment techniques: (i) adversarial domain adaptation (ada) @xcite , (ii) domain-adaptive fine-tuning (daft) @xcite , and (iii) a new instance weighting scheme using language model likelihood scores (liw). our results show that daft and liw improve over bert on both domains, whereas ada only improves on notes. across domains, ada and daft perform best on notes and conversations respectively. to probe why some techniques are better at addressing certain source-target domain pairs, we analyze model performance on various types of covariate shifts (e.g., lexical shift, semantic shift). our analysis uncovers interesting patterns such as varying ability of models to leverage subword morphology to generalize to technical terms, and liw's performance improvement on long-term state events (e.g., chronic illnesses). our best models achieve f1 scores of 70.0 and 72.9 on notes and conversations respectively with no training data. foot_0 2 related work ## dataset creation to test adaptability of event extraction models, we create a testbed using data from two domains: 1. clinical notes: clinical notes are records documenting physician observations from their interactions with patients. they usually detail various aspects of a patient's care such as present illness, symptoms, medical history, treatments, and test results. they share a thematic structure, though particular specialties (e.g., cardiology) and institutions often incorporate their own modifications. we collected a set of 4999 de-identified clinical notes from 40 specialties, by scraping mtsamples. 3 the notes are reference samples provided by various users, with names and dates edited for confidentiality. they are freely available to print, share, link and distribute, as per website policy. average length of a clinical note is 652 tokens. 2. doctor-patient conversations: this data contains human-transcribed, de-identified conversations recorded during physician-patient visits. the conversations often follow a similar schema, with patients describing their symptoms, doctors inquiring about ongoing treatments, and then suggesting potential follow-up treatments/tests. we use a proprietary database of 63,540 conversations covering 53 specialties, collected by abridge ai inc. physicians across a variety of specialties are contracted to record natural in-office conversations with their patients who agree to participate in the research by providing verbal and written consent. recordings are made on a digital recording device or a smartphone application and are uploaded to a secure server where they are scrubbed of all identifiable information, in accordance with the health insurance portability and accountability act of 1996 (hipaa) privacy rule. de-identified recordings are transcribed and stored in a database, which currently contains over 100,000 recordings dating from 2006 to 2017. average conversation transcript length is 2309 tokens. these domains exhibit different types of linguistic shifts from the source (news). while both domains exhibit a shift in vocabulary, it is more pronounced in clinical notes since they are written by doctors (experts) who use highly technical terms. conversely, shifts in syntax are more pronounced in conversations due to the prevalence of repetition, back-channeling, interruptions etc. semantic shifts are more pronounced in conversations since they contain a higher proportion of hypothetical statements (e.g., when doctors ask questions, make requests or "think out loud") than both notes and news articles which tend to serve as records of actual events. to better evaluate model performance on linguistic shifts, we control for topical variation across domains by limiting our focus to 3 specialties: cardiovascular/pulmonary (cardio), obstetrics/gynaecology (obgyn) and hematology/oncology (onco). these specialties are well-represented in both notes and conversations, and cover events with a variety of temporalities ranging from intervals with fixed duration (e.g., pregnancy), to intervals with indeterminable endpoints (e.g., long-term cardiac failure). ## methods for marginal alignment to adapt event extraction models with no training data, we tackle the problem of covariate shift, which arises when the marginal distribution (or input distribution) p (x) changes between train and test data. directly applying a supervised model trained on the training set, to the test set might not statistic news notes convos ## analysis and discussion tables 4 and 5 provide an indication of model ability to handle covariate shift. however, covariate shift occurs at multiple layers in language (e.g., lexical level, syntactic level, etc.), leading to different dimensions of variation between domains (e.g., topical variation, genre variation, etc.). looking at overall model performance does not offer insight into whether there are specific shifts that some models are better at addressing. we dig deeper into this question, focusing on two levels of shift: (i) lexical shift, and (ii) semantic (event type) shift. variation under lexical shift: we separate model performance on in-vocabulary (iv) and out-ofvocabulary (oov) tokens. note that the proportion of events that are oov is higher in clinical notes (52%) than conversations (20.6%). tables 6 and 7 present model performance on these token categories. surprisingly, despite the use of specialized language, oov performance on clinical notes is higher than conversations for all models except bert-daft. taking a closer look at the oov event instances from clinical notes that models identify correctly, we see that a large proportion (54.8%) contain one of three morphological patterns: (i) past tense verbs ending in "-ed", (ii) gerunds ending in "-ing", or (iii) nouns ending in "-tion" or "-sion". these patterns are also common among events in the news domain. for example, past tense verbs often refer to events that have already occurred and gerunds and nouns ending in "-tion" refer to processes. we hypothesize that bert-based models might be exploiting these morphological regularities to correctly label unseen medical terms (e.g., irrigated, excision, dissected, wheezing, etc.). these patterns are more prevalent in notes (35.6%) than conversations (23.5%), explaining the surprising performance difference. variation under semantic shift: to determine whether model performance on oov tokens depends on event type, we randomly sample ∼500 oov tokens from each domain and label them for event type. we use the same typology as timeml (state, i-state, occurrence, aspectual, reporting, perception, i-action, none), with additional labels for the event types we introduce (activitypattern, longtermstate). 9 we run an anova model with each token per model as an instance (total 5080 instances), noting event type, target (notes/convos), model (bert/ada/liw/daft/daft-syn) and correctness (1 vs 0). correctness is the dependent variable, while others are independent variables. we include all pairwise interaction terms and the three way interaction between event type, target and model. we see a positive main effect of event type on correctness (@xmath0.0001), indicating that some event types are more difficult. there are two significant two-way interactions, one between target and event type (@xmath1.0001), indicating that difficulty of event types differs across sources, and between model and event type (@xmath2.0001), indicating that which model is better depends on event type. three way interaction between model, event type, and target is also significant (@xmath3.0001), indicating that performance differences between models per event type differs between sources. we interpret differences in performance per event type separately for each source using a student-t post-hoc analysis to determine which pairwise contrasts are statistically significant. this reveals that in clinical notes, liw outperforms all models on i-state events (i.e., hypothetical, future or negated states) and longtermstate events, a category never seen in the training data. these improvements might stem from the training algo-rithm used by liw. liw up-weights instances in news that resemble medical data, which contains a high proportion of these event categories. therefore, despite being infrequent in news, they get up-weighted, helping liw identify them better. ## conclusion in this work, we focused on unsupervised adaptation of event extractors to new domains by aligning the marginal distributions of source and target domains. we created two event extraction test sets using english texts from two medical domains: (i) clinical notes, and (ii) doctor-patient conversations, and tested the efficacy of three alignment techniques: (i) adversarial domain adaptation (ada), (ii) domain adaptive fine-tuning (daft), and (iii) a new instance weighting technique based on language model likelihood scores (liw). none of these models consistently outperformed the others, but a deeper analysis of model performance under different types of shifts (e.g., lexical shift, semantic shift) uncovered interesting variations among models. our best-performing models attained f1 scores of 70.0 and 72.9 on notes and conversations respectively, using no labeled target data. we believe these models define a good starting point and can be further improved using few-shot learning.