Domain adaptation in practice: Lessons from a real-world information extraction pipeline (2021)

Abstract: Advances in transfer learning and domain adaptation have raised hopes that once-challenging NLP tasks are ready to be put to use for sophisticated information extraction needs. In this work, we describe an effort to do just that – combining state-of-the-art neural methods for negation detection, document time relation extraction, and aspectual link prediction, with the eventual goal of extracting drug timelines from electronic health record text. We train on the THYME colon cancer corpus and test on both the THYME brain cancer corpus and an internal corpus, and show that performance of the combined systems is unacceptable despite good performance of individual systems. Although domain adaptation shows improvements on each individual system, the model selection problem is a barrier to improving overall pipeline performance.

## introduction advances in machine learning methods and the release of annotated datasets of clinical texts @xcite in the past decade has led to an increase of available clinical nlp systems for interesting tasks. recent advances in pre-trained models @xcite have made ever more accurate clinical nlp systems possible. unsupervised domain adaptation algorithms (e.g., @xcite ) have made it possible to reduce performance degradation when applying trained models to new domains. the great promise of these developments is that these methods can be combined into pipelines that allow for sophisticated information extraction capabilities for downstream clinical use cases. rather than building one-off datasets for each complex downstream task that arises, standard nlp components could potentially be used as "lego"-style building blocks that allow for flexibly approaching new tasks as they arise. however, the existence of the building blocks alone does not solve this problem. combining individual components into nlp pipelines can lead to cascading errors @xcite . the true error rate for structured extraction tasks is potentially as high as the sum of the component tasks' errors. for example, if the goal is to extract normalized concepts with assertion status, the concept error can come from normalization error, negation detection error, uncertainty detection error, etc, and the errors may not be correlated. these problems are exacerbated in the common case where individual components are trained on data from different domains, and tested on data from yet another domain. in this work, we quantitatively examine the issues described above in the context of extracting drug temporality signatures, with the goal of understanding drug start and stop events. we approach this task with the combination of three sub-tasks: 1) the temporal relation of these mentions to the document creation time (doctimerel), 2) negation status of the mention, and 3) aspectual link relations of the mention (e.g., is it being described as starting or stopping). figure 1 shows an example sentence with a drug mention, that demonstrates how the three tasks work together to establish the status of that drug in that patient. successfully solving this task is beneficial for understanding patient treatment course, and enabling more causal understanding in important tasks such as adverse drug event detection or relating medication courses to outcomes. we first set state-of-the-art benchmarks for three tasks on the thyme corpus by fine-tuning large pre-trained transformer models @xcite . we then examine how the performance of individual systems degrades when moving from the training data to our target data (a pediatric cardiology cohort), and how the overall system performs when combining multiple systems additionally , as the patient has preserved ejection fraction , no prior history of embolic phenomena , and no significant valvular disease , it would be acceptable for him to remain off e coumadin /e for the interim . figure 1 : an example sentence with highlighted drug name coumadin to be classified for all three tasks. the gold standard has this drug mention classified as negated, with @xmath0, and @xmath1. these three facts can be used to understand that the patient is not on the drug now or going forward, and was likely not on the drug prior to the note as well. with imperfect performance. despite strong individual results, we find that performance suffers immensely due to both out-of-domain performance losses and the basic combinatorial math of integrating outputs from multiple systems. this is the case even though we use a metric, accuracy, that is forgiving to the worst-performing individual model. ## background it is both formally and empirically understood that classifiers can suffer performance loss when the test data is drawn from a different distribution than the training data (sometimes called domain shift). this presents a difficult challenge in clinical nlp because data-sharing limitations make it difficult to create large and diverse training corpora. as a result, domain adaptation approaches have been applied to multiple tasks in clinical nlp @xcite @xcite . recent work in the general domain has made use of transfer learning, which can attack the problem of domain shift, but by a different mechanism than domain adaptation; by training on massive corpora, large pre-trained models both learn general features, and are able to learn from smaller new datasets without overfitting. the most prominent of these models are based on the transformer architecture @xcite . bert @xcite uses a transformer encoder, and has shown that pre-training with massive amounts of text on a language modeling task, then fine-tuning on a supervised task of interest, achieves large performance gains in multiple nlp tasks. 1 during fine-tuning for sentence classification tasks, a classification head with randomly initialized weights is attached to a special sentenceinitial token. fine-tuning then proceeds in a standard supervised learning paradigm, with the goal of learning the weights of the classification head, but where the weights of all of the transformer encoder layers can also be updated. we use robertabase, a 12-layer transformer encoder that provides excellent performance but manageable memory utilization for our hardware @xcite . the bigger vision of our current work is extracting temporally-aware medication mentions from electronic health records data. this would enable important downstream tasks including automatically extracting drug timelines to correlate with treatments, or extracting better causal information about drugs and potential adverse events. some other recent work has also examined this topic @xcite , but focused on a single drug class (proton pump inhibitors), was limited to the problem list section, and made the assumption that missing drug implied drug stoppage. ## methods we began this work by developing several nlp components necessary to extract drug temporality signatures, including negation detection, relation to document creation time (doctimerel), and aspectual link extraction (alink), all detailed below. detecting negation helps us avoid false positives from mentions corresponding to, for example, decisions to not use a drug. doctimerel helps us distinguish mentions of drugs that are current from those that predate the current time period, or are being speculated about for future use. alink can model drug start, stop, and continuation events, which can help to distinguish whether a missing mention in the middle of a record corresponds to a stop and restart, or an incidentally omitted mention. figure 1 shows an example instance of a drug mention to be classified for all three tasks. the thyme dataset @xcite , released as part of clinical tempeval @xcite , contains all three of these annotation types, on 1200 notes of patients with colon and brain cancer. we train all models on the colon cancer section (details on data are in section 4). while our bigger project is specific to drug mentions, the problem is not limited to drug mentions, so we train and evaluate on all annotated events in the thyme corpus. we also assume that events are given, to allow a straightforward metric of how many events we "get right" when combining all property predictions. in the real world, events will have to be automatically detected, so our metric will be an upper bound on how often the combined models get everything correct. ## evaluation for the three tasks of interest, we evaluate indomain (thyme colon cancer corpus), as well as one closely related out-of-domain corpus (thyme brain cancer corpus). we also use a second out-ofdomain corpus, an internal data set we annotated for all three tasks (pulmonary hypertension [ph] notes). this annotation was performed by an experienced annotator who has worked on clinical annotation projects in the past. we measure performance on negation with f1score, on doctimerel with accuracy (because the classes are relatively balanced), and on alink extraction with the average f1 score of all categories, macro-f1 (because the high frequency none label makes accuracy uninformative). in addition to system-level performance, we report an evaluation of mention-level accuracy: an event is counted as correct if all three systems made the correct prediction, and we report the percentage of events that were correct. this setting estimates how usable the entire pipeline is, given different system settings. the "colon" columns of the "brain" and "ph" columns of mention level accuracy ("all" column) is good for the in-domain data (thyme colon cancer), but drops off substantially even for the thyme brain cancer corpus from the same institution, created with the same guidelines and using the same annotators. the mention level accuracy for our internal ph data is unusable at an accuracy of 0.506 with roberta+lm. this accuracy means that roughly one of every two drug mentions will have at least one of its attributes classified incorrectly. ## discussion and conclusion the results also show that combining nlp systems for new, complex, information needs is likely to run into issues even when individual systems perform well. in particular, our experiments raise questions about real-world use of domain adaptation. if we treated thyme colon and brain sets as representative in-domain and out-of-domain datasets we would select roberta or roberta+lm for everything. but an oracle optimizing ph performance would tell us to use the svm for negation and doctimerel and roberta+lm for alink. one of the difficulties in even studying domain adaptation is model selection -if labeled target data is not available, standard practices like tuning on held out data are impossible. but the reality our results suggest is that different algorithms work well on different tasks and datasets, and selecting the best model for each task is an unsolved and under-studied problem. one direction of research that may address these concerns is on better modeling of domains themselves. the problem has been exacerbated with the move from feature-based classifiers to pre-trained black box models, as it is now even more difficult to understand the cause of errors in new domains without interpretable features. domain adaptation should leverage "bertology" and interpretability research to help understand how different aspects of domains contribute to performance differences. for example, in clinical notes, variation in institutions, specialties, note types, authors, etc. probably contribute differently to domain shift, and these sources of variation should be empirically explored. future work will explore this direction to develop unsupervised model selection algorithms that better predict target domain performance.