A BERT -based One-Pass Multi-Task Model for Clinical Temporal Relation Extraction (2020)

Abstract: Recently BERT has achieved a state-of-the-art performance in temporal relation extraction from clinical Electronic Medical Records text. However, the current approach is inefficient as it requires multiple passes through each input sequence. We extend a recently-proposed one-pass model for relation classification to a one-pass model for relation extraction. We augment this framework by introducing global embeddings to help with long-distance relation inference, and by multi-task learning to increase model performance and generalizability. Our proposed model produces results on par with the state-of-the-art in temporal relation extraction on the THYME corpus and is much “greener” in computational cost.

## introduction the analysis of many medical phenomena (e.g., disease progression, longitudinal effects of medications, treatment regimen and outcomes) heavily depends on temporal relation extraction from the clinical free text embedded in the electronic medical records (emrs). at a coarse level, a clinical event can be linked to the document creation time (dct) as document time relations (doctimerel), with possible values of before, after, over-lap, and before overlap @xcite . at a finer level, a narrative container @xcite can temporally subsume an event as a contains relation. the thyme corpus @xcite consists of emr clinical text and is annotated with time expressions (timex3), events (event), and temporal relations (tlink) using an extension of timeml @xcite . it was used in the clinical temp-eval series @xcite @xcite . while the performance of doctimerel models has reached above 0.8 f1 on the thyme corpus, the contains task remains a challenge for both conventional learning approaches @xcite @xcite and neural models (structured perceptrons @xcite , convolutional neural networks (cnns) @xcite , and long short-term memory (lstm) networks @xcite @xcite ). the difficulty is that the limited labeled data is insufficient for training deep neural models for complex linguistic phenomena. some recent work @xcite has used massive pre-trained language models (bert; @xcite and their variations @xcite for this task and significantly increased the contains score by taking advantage of the rich bert representations. however, that approach has an input representation that is highly wasteful -the same sentence must be processed multiple times, once for each candidate relation pair. inspired by recent work in green ai @xcite , and one-pass encodings for multiple relations extraction @xcite , we propose a one-pass encoding mechanism for the contains relation extraction task, which can significantly increase the efficiency and scalability. the architecture is shown in figure 1 . the three novel modifications to the original one-pass relational model of @xcite are: (1) unlike @xcite , our model operates in the relation extraction setting, meaning it must distinguish between relations and nonrelations, as well as classifying by relation type. for the contains task, we create relation candidates from all pairs of entities within an input sequence. each candidate is represented by the concatenation of three embeddings, e i , e j , and g, as [g:e i :e j ], where g is an average-pooled embedding over the entire sequence, and is different from the embedding of [cls] token. the [cls] token is the conventional token bert inserts at the start of every input sequence and its embedding is viewed as the representation of the entire sequence. the concatenated embedding is passed to a linear classifier to predict the contains, contained-by, or none relation, rij , as in eq. ( 1 ). similarly, for the doctimerel (dtr) task we feed each entity's embedding, e i , together with the global pooling g, to another linear classifier to predict the entity's five "temporal statuses": timex if the entity is a time expression or the dtr type (before, after, etc.) if the entity is an event: for the combined task, we define loss as: ## discussion through combining mtl with a one-pass mechanism produces a more efficient and generalizable model. with merely additional 15k flops ( in conclusion, we built a "green" model for a challenging problem. deployed on a single gpu with 25 times better efficiency, it succeeded in both temporal tasks, achieved better generalizability, and suited to other pre-trained models @xcite @xcite @xcite .) #1: subsequent after staging will include after mri of the pelvis... #2: his colon cancer before screening in the past has been fecal occult blood before/overlap tests yearly since the age of 55...