Self-Supervised Detection of Contextual Synonyms in a Multi-Class Setting: Phenotype Annotation Use Case (2021)

Abstract: Contextualised word embeddings is a powerful tool to detect contextual synonyms. However, most of the current state-of-the-art (SOTA) deep learning concept extraction methods remain supervised and underexploit the potential of the context. In this paper, we propose a self-supervised pre-training approach which is able to detect contextual synonyms of concepts being training on the data created by shallow matching. We apply our methodology in the sparse multi-class setting (over 15,000 concepts) to extract phenotype information from electronic health records. We further investigate data augmentation techniques to address the problem of the class sparsity. Our approach achieves a new SOTA for the unsupervised phenotype concept annotation on clinical text on F1 and Recall outperforming the previous SOTA with a gain of up to 4.5 and 4.0 absolute points, respectively. After fine-tuning with as little as 20% of the labelled data, we also outperform BioBERT and ClinicalBERT. The extrinsic evaluation on three ICU benchmarks also shows the benefit of using the phenotypes annotated by our model as features.

## introduction supervised fine-tuning on the top of the bertbased models has recently become the standard approach in information extraction delivering stateof-the-art (sota) results across different tasks @xcite . the dependence of these models on the availability of the costly human-annotations remains a serious obstacle towards a large scale deployment of such models. this problem is especially actual in the clinical domain with limited availability of experts. in the self-supervised setting, automatic annotations are cheap to produce. some rule-based automatic labelers, such as chexpert @xcite , which is built on negbio @xcite , are often used to create training data for supervised bert-based models (e.g., automatic annotators of radiology reports @xcite ). those models usually generalise over a small set of classes (under 20). other automatic labelers exploit ontologies. for example, the umls (unified medical language system) ontology @xcite is almost predominantly used to match linguistics patterns in clinical text to medical concepts (e.g., using the metamap tool @xcite ). due to the complexity of the information extraction task in this challenging setting (sparse multi-class), the approaches that use the data annotated (e.g., @xcite @xcite ) mostly rely on non-contextualised embeddings focusing on the detection precision. however, especially for clinical text, which is noisier and exhibits a variety of clinical expressions requiring disambiguation, relying on the context is essential. we argue that recall is very important, especially when automatic annotation results are used further in the downstream tasks. in this work, we propose a self-supervised approach for sparse multi-class classification that fully relies on the context to detect contextual synonyms of medical concepts in clinical text. to be more precise, our model is based on the clinical-bert @xcite model which was pre-trained on the biomedical and clinical corpora that are widely used, producing state-of-the-art results in a range of supervised biomedical tasks, e.g. named entity recognition, relation extraction and question answering @xcite . we separate the detection of frequent and rare classes by introducing different training objectives. the special training objective for rare classes increases the proximity of the respective textual embeddings and the ontology embeddings of concepts. our work also exploits data augmentation techniques, such as paraphrasing and guided text generation to aid sparse class detection and diversify the training data. we apply our methodology for the phenotype detection task with more than 15,000 concepts from the human phenotype ontology (hpo) @xcite foot_0 . the phenotyping task is an important clinical nlp task that can improve the understanding of disease diagnosis @xcite @xcite @xcite . it remains unexplored due to the complexity of the classification into that large amount of classes. we test our approach on clinical data, namely on electronic health records (ehrs) and radiology reports. our main contributions: (1) self-supervised methodology for contextual phenotype detection in clinical records. (2) methodology for sparse class detection with the special training objective that increases proximity of contextual synonyms to ontology embeddings. (3) data augmentation methodology to further improve the detection of sparse classes. our self-supervised models improve the current sota on f1 up to 4.5 absolute points, while on recall up to 4.0 absolute points for the phenotype detection task for clinical data, which demonstrates how relying on the context is essential for this type of data. second, after fine-tuning, our model outperforms the fine-tuned bert-based models with as little as 20% of labelled data, which confirms efficiency of our self-supervised training objectives. moreover, the extrinsic evaluation shows the benefits of using the phenotypes annotated by our model as features to predict icu patient outcomes. we present related work in section 2, our phenotyping methods in section 3, and our experimental setup in section 4. then, we present and discuss key results in section 5. finally, we conclude this work in section 6. ## related work most of the current methodologies for phenotype detection are supervised, bert-based (e.g., biobert @xcite or clinical-bert @xcite ) and dedicated to the detection of certain rather limited phenotypes or their groups @xcite @xcite @xcite . unsupervised methods in the clinical nlp domain traditionally rely on the usage of ontologies and knowledge bases. human phenotype ontology @xcite is the most widely used ontology of phenotypes. the use of hpo in annotating phenotypic information automatically remains unexplored, mainly due to the complexity of formalising the task with over 15,000 concepts. such methods as metamap @xcite (the mayo clinic tool @xcite based on it), ctakes @xcite , ncbo @xcite and clinphen (deisseroth et al., 2019) follow similar pipelines and use linguistics patterns for shallow matching. more recently, unsupervised deep learning methods have been applied to the problem, which allowed to perform the semantic analysis and go beyond shallow matching @xcite @xcite . these approaches use non-contextualised embeddings, focus on the precision of detection with limited context exploitation. for example, the authors in @xcite propose a procedure to learn vectors of words enriched with their averaged context over the corpus to map them to correct medical concepts. we use contextualised word representations in contrast to all the related approaches and focus on recall. ## methodology this section introduces the problem of phenotype detection along with our self-supervised method. it elaborates our data augmentation strategies, selective supervision in low-resource conditions, and finally explains our inference algorithm. problem definition while annotating clinical text, clinicians usually relate hpos to short spans, which usually have around 2-3 words depending on the corpus. 3 following this rationale, we define the phenotype annotation as a two-step process: (1) detect hpo-relevant text spans, and (2) assign respective hpo concepts to those spans. more formally, given a textual document @xmath0, ..., t n } represented by a sequence of tokens, and a full set of transformer encoder patient shows abdominal pain 𝑣 ! ! 𝑣 ! " 𝑣 ! # 𝑣 ! $ classifier 1 classifier 2 objective 1 binary classification hpo / non hpo objective 2 cross entropy for frequent hpos objective 3 euclidean distance for rare hpos transformer encoder data augmentation real clinical textual data pooling 𝑣 " % unpleasant sensation discomfort … definition of an hpo concept (e.g., abdominal pain hp:0002027) hpo embeddings trained in separate by objective 4: relational loss objective 5: semantic loss 𝑣 " ! 𝑣 " & … a branch of hpo hpo concepts @xmath1, ..., h m } under the root node phenotypic abnormality (hp:0000118) (exclusive) of the hpo ontology, our goal is to model: in the self-supervised setting, we consider only the training examples with textual spans matched with exact match to the hpo concepts as defined by the ontology. the main assumption here is that by capturing context of those term spans, the model will be able to generalise and detect formally different hpo spans seen in the similar contexts as the hpo concepts (a.k.a. contextual synonyms, for example, fever (hp:0001945) will be matched to "feverish"). to support this challenging setting, we have designed a series of relevant training objectives described below. training objectives as shown in figure 1 (left), the proposed model for phenotype annotation consists of a transformer encoder which is identical to and initialised with clinicalbert @xcite . besides, there are two additional classifiers on the top of the transformer encoder which predict if a token is hpo-relevant and assign hpo concepts to those hpo-relevant tokens. we enrich the model with the following three training objectives. (1) a binary cross-entropy loss l 1 to predict p(1 h |t n ), where 1 h is 1 if t n is hpo-relevant, otherwise 0. (2) a cross-entropy loss l 2 with softmax to predict p(h m |t n ), which is defined over the most frequent hpo concepts found in the training data. the intuition behind this objective is to increase precision of prediction in the resulting performance. (3) the euclidean distance l 3 between the token embedding v tn and the respective hpo concept embedding v hm , which is defined to increase recall of the model and targets the detection of the rare hpo concepts. note that the objectives above can be used for pre-training and further fine-tuning of the models in a way similar to bert. ## experimental setup this section will introduce the datasets, implementation details, baselines and evaluation metrics. ## results and discussion this section discusses the results for the selfsupervision and selective supervision settings. we note that our method does not give better performance for the pubmed dataset. we hypothesise that this happens due to the difference of gold annotation standards, as well as the fact that this dataset is oriented towards the detection of rare phenotypes with less frequent context patterns that are hence difficult to learn for our model. ## conclusion in this paper, we have proposed a deep selfsupervised phenotype annotation approach relying on contextualised word embeddings and data augmentation techniques. our experimental results in a challenging sparse multi-class setting, with over 15,000 candidate hpo concepts, indicate that our methodology is particularly efficient to detect contextual mentions of phenotype concepts in clinical text. we demonstrate that increasing phenotyping recall is essential for downstream tasks.