Detect and Classify – Joint Span Detection and Classification for Health Outcomes (2021)

Abstract: A health outcome is a measurement or an observation used to capture and assess the effect of a treatment. Automatic detection of health outcomes from text would undoubtedly speed up access to evidence necessary in healthcare decision making. Prior work on outcome detection has modelled this task as either (a) a sequence labelling task, where the goal is to detect which text spans describe health outcomes, or (b) a classification task, where the goal is to classify a text into a predefined set of categories depending on an outcome that is mentioned somewhere in that text. However, this decoupling of span detection and classification is problematic from a modelling perspective and ignores global structural correspondences between sentence-level and word-level information present in a given text. To address this, we propose a method that uses both word-level and sentence-level information to simultaneously perform outcome span detection and outcome type classification. In addition to injecting contextual information to hidden vectors, we use label attention to appropriately weight both word and sentence level information. Experimental results on several benchmark datasets for health outcome detection show that our proposed method consistently outperforms decoupled methods, reporting competitive results.

## introduction access to the best available evidence in context of patient's individual conditions enables healthcare professionals to administer optimal patient care @xcite . healthcare professionals identify outcomes as a fundamental part of the evidence they require to make @xcite . williamson et al. * danushka bollegala holds concurrent appointments as a professor at university of liverpool and as an amazon scholar. this paper describes work performed at the university of liverpool and is not associated with amazon. sentence there were no significance betweengroup differences in the incidence of wheezing or shortness of breath (2017) define an outcome as a measurement or an observation used to capture and assess the effect of treatment such as assessment of side effects (risk) or effectiveness (benefits). with the rapid growth of literature that reports outcomes, researchers have acknowledged and addressed the need to automate the extraction of outcomes from systematic reviews @xcite and answering clinical questions @xcite . @xcite mention that automated health outcomes detection (hod) could speed up the process of analysing and assessing the effectiveness of clinical interventions in evidence based medicine (ebm; @xcite . hod has been conducted in the past as either an outcome span detection (osd) task, where we must detect a continuous span of tokens indicating a health outcome @xcite or as an outcome classification (oc) task, where the goal is to classify the text spans into a pre-defined set of categories @xcite @xcite . however, the two tasks are highly correlated and local token-level information enables us to make accurate global sentence-level outcome predictions, and vice versa. an outcome type predicted for a text span in a sentence must be consistent with the other outcome spans detected from the same sentence, while the outcome spans detected from a sentence must be compatible with their outcome types. these mutual compatibility constraints between outcome spans and their classes will be lost in a decoupled approach, resulting in poor performance for both osd and oc tasks. two illustrative examples in health outcome detection (hod): given a sentence @xmath0, . . . , w m extracted from a clinical trial abstract, the goal of hod is to identify an outcome span o @xmath1, . . . , b n (i.e osd), and subsequently predict a plausible outcome type t(o d ) ∈ y for o d (i.e. oc), where 1 ≤ @xmath2, and y is a predefined set of outcome types. we propose label context-aware attention model (lcam), a sequence-to-sequence-to-set (seq2seq2set) model, which uses a single encoder to represent an input sentence and two de-coders -one for predicting the label for each word in osd and another for predicting the outcome type in oc. lcam is designed to jointly learn contextualised label attention-based distributions at word-and sentence-levels in order to capture which label/s a word or a sentence is more semantically related to. we call them contextualised because they are enriched by global contextual representations of the abstracts to which the sentences belongs. ## related work joint training to achieve a dichotomy of tasks has previously been attempted, particularly for sequence labelling and sentence classification. targeting named entity recognition (ner) and relation extraction (re), chen et al. (2020) transfer bert representations via a joint learning strategy to extract clinically relevant entities and their syntactic relationships. in their work, the joint learning models exhibit dramatic performance improvements over disjoint (standalone) models for the re task. our work differs from (chen et al., 2020) in that we use attention layers prior to the first and second classification layers. ma et al. (2017) train a sparse attention-based lstm to learn context features extracted from a convolution neural network (cnn). the resulting hidden representations are used for label prediction at each time step for sequence labelling, and subsequently aggregated via average pooling to obtain a representation for sentence classification. the sparse constraint is strategically biased during weights assignment (i.e. important words are assigned larger weights compared to less important words). karimi et al. (2020) perform absa @xcite by feeding a bert architecture with a sentence @xmath3, x 1:j , [sep ], x j+1:n , [sep ]), where x 1:j is a sentence containing an aspect of a product, x j+1:n is a customer review sentence directed to the aspect and [cls] is a token not only indicating the beginning of a sequence, but also a sentiment polarity in the customer review about the aspect. they fine-tune a bert model to conduct both aspect extraction and aspect sentiment classification. the above mentioned works tend to generate attention-based sentence-level representations that encapsulate the contribution each word would make in predicting sentence categories. we however generate label-inclined attention representations at word-level that can be used to effectively deduce word categories/labels. to the best of our knowledge, we are the first to perform a joint learning task that achieves mlp at two classification stages, token-and sentence-levels, while using only the top predictions at token level. ebm-comet ebm-nlp ebm-comet + ebm-nlp # of abstracts 300 5000 5300 # of sentences 5193 40092 45285 # of outcome labels 5 6 5 avg sentence length 21.0 26.0 25.0 # of training sentences 4155 32074 36229 # of testing sentences 1038 8018 9056 motivated by the taxonomy proposed by @xcite to standardise outcome classifications in electronic databases and inspired the annotation of ebm-comet dataset @xcite , we attempt to align ebm-nlp's arbitrary outcome classifications to standard outcome classifications that are proposed by @xcite . these standard classifications were found (after extensive analysis and testing) to provide sufficient granularity and scope of trial outcomes. we propose an unsupervised label alignment method to identify and align parallel annotations across the ebm-nlp and ebm-comet. additionally, we use the discovered semantic similarity between the two datasets and merge them in order to create a larger dataset for evaluating our joint learning approach. the merged dataset contains labels that follow the taxonomy proposed by @xcite . all three datasets are used during evaluation, with each one being randomly split into two, where 80% is retained for training and 20% for testing as shown in for this purpose, we first create an embedding for each label l s in a sentence s(∈ s) by applying mean pooling over the span of embeddings (extracted using pre-trained biobert @xcite ) for the tokens corresponding to an outcome annotated with l s as shown in (1). next, we average the embeddings of all outcome spans that are annotated with l s in all sentences in s to generate an outcome type label embedding l s . likewise, we create an outcome type label embedding, l t for each outcome type in the target dataset t . after generating label embeddings for all outcome types in both s and t , we compute the cosine similarity between each pair of l s and l t as the alignment score between each pair of labels l s and l t respectively. results show that physiological outcomes (containing domain p 0) are similar to physical outcomes and therefore the latter outcomes are labelled physiological, life-impact outcomes are similar to mental outcomes and therefore the latter outcomes are labelled life-impact. mortality and adverse-effects outcomes both remain unchanged because both categories exists in source and target datasets, and their respective outcomes are discovered to be similar. we evaluate the lcam architecture on the resulting merged dataset, and additionally, evaluate the alignment approach by comparing the performances before and after merging. ## conclusion we proposed a method to jointly detect outcome spans and types using a label attention approach. moreover, we proposed a method to align multiple comparable datasets to train a reliable outcome classifier. given real-world scenarios where it is often impractical or computationally demanding to build a model for each and every single task, our experimental results demonstrate the effectiveness of an approach that simultaneously (jointly) achieves two different task without compromising the performance of the individual tasks when decoupled.