Extracting Chemical-Protein Interactions via Calibrated Deep Neural Network and Self-training (2020)

Abstract: The extraction of interactions between chemicals and proteins from several biomedical articles is important in many fields of biomedical research such as drug development and prediction of drug side effects. Several natural language processing methods, including deep neural network (DNN) models, have been applied to address this problem. However, these methods were trained with hard-labeled data, which tend to become over-confident, leading to degradation of the model reliability. To estimate the data uncertainty and improve the reliability, “calibration” techniques have been applied to deep learning models. In this study, to extract chemical–protein interactions, we propose a DNN-based approach incorporating uncertainty information and calibration techniques. Our model first encodes the input sequence using a pre-trained language-understanding model, following which it is trained using two calibration methods: mixup training and addition of a confidence penalty loss. Finally, the model is re-trained with augmented data that are extracted using the estimated uncertainties. Our approach has achieved state-of-the-art performance with regard to the Biocreative VI ChemProt task, while preserving higher calibration abilities than those of previous approaches. Furthermore, our approach also presents the possibilities of using uncertainty estimation for performance improvement.

## introduction in the biomedical domain, there exist several entities, such as genes, chemicals, and diseases, that are closely related to each other. therefore, extracting the relationships among these entities is critical for biomedical research, particularly in fields such as construction of a knowledge base or drug development. biomedical text data, including pubmed abstracts, usually contain information about biomedi-cal entities and their relationships with each other. thus, various natural language processing models, particularly deep learning models, are applied to biomedical text data to extract the relationships among these entities, as a kind of classification task. chemprot corpus @xcite is the first corpus dataset for chemical-protein (gene) relationship extraction, which has been conducted by biocreative vi organizers. these organizers annotated all entity offsets of chemical and protein mentions and relationship types between chemicals and proteins (chemical-protein relations, cpr). there exist 10 groups of the relationship types, and five of these (cpr:3, cpr:4, cpr:5, cpr:6, and cpr:9) were used in the evaluation. all models for extracting relationships from chemprot data are designed as classifiers. in a deep learning-based multi-class classifier, the output probability distribution for each class is calculated through the softmax function. in the training step, the model is trained to maximize the output probability of the correct class. however, some studies reported that the deep learning classifier trained with hard-labeled data (1 for correct class, 0 for else) tends to become over-confident @xcite . this overconfidence does not directly affect classification performance, but it degrades the reliability of the model. in other words, the output probability of the over-confident model does not indicate how uncertain the input example is, even if its classification performance is high. therefore, several approaches, called "calibration" techniques, have been applied to several domains that require high reliability, such as autonomous driving and medical diagnosis @xcite . in the natural language processing domain, bidirectional encoder representation from transformers (bert) @xcite was proposed for a wide-range of language understanding. bert is a large multi-head attention @xcite model, which was pre-trained with a vast amount of corpus data. this pre-trained model can be easily transfer-learned and can be applied on several downstream tasks (e.g. sentence classification) by fine-tuning it. bert has been used in many domains, including a biomedical field. nevertheless, it is still important to improve the performance of bert by applying additional techniques while using the bert as a backbone architecture. in this study, we propose a dnn-based approach to improve the performance of chemical-protein relationship extraction, while calibrating the classifier. more precisely, we incorporated two main calibration techniques to bert @xcite to improve the reliability and performance. furthermore, we propose a semi-supervised learning workflow using the calibrated model and unlabeled in-domain data. the main contributions of our study are as follows: ## conclusion in this study, we propose a calibrated deep neural network-based relationship extraction model for chemical-protein interactions. we applied mixup training-which can both augment the training examples and calibrate the model-on the biobert model. we also incorporated the low entropy penalizing term in the loss function, as a regularization term during training. this led to significant improvement in terms of both classification and calibration performance. moreover, we applied self-training on our model to augment the training data and boost the performance, as our model is well-calibrated; it returned reliable output probabilities. consequently, our model outperformed the other chemical-protein relationship extraction models and achieved state-of-the-art performance regarding the biocreative vi chemprot task. in this work, we applied our training process on the biomedical domain, especially on chemicalprotein relation extraction, because a large set of unlabeled data can be found from pubmed in the biomedical domain. this process can be applied to any nlp classification problems with unlabeled data sets. we can apply the proposed method to other tasks such as cola or sst-2 in the glue benchmark @xcite since there are a large set of unlabeled data for ungrammatical sentences or movie reviews. the application of our method to other domains will be our future work.