View Distillation with Unlabeled Data for Extracting Adverse Drug Effects from User-Generated Data (2021)

Abstract: We present an algorithm based on multi-layer transformers for identifying Adverse Drug Reactions (ADR) in social media data. Our model relies on the properties of the problem and the characteristics of contextual word embeddings to extract two views from documents. Then a classifier is trained on each view to label a set of unlabeled documents to be used as an initializer for a new classifier in the other view. Finally, the initialized classifier in each view is further trained using the initial training examples. We evaluated our model in the largest publicly available ADR dataset. The experiments testify that our model significantly outperforms the transformer-based models pretrained on domain-specific data.

## introduction social media has made substantial amount of data available for various applications in the financial, educational, and health domains. among these, the applications in healthcare have a particular importance. although previous studies have demonstrated that the self-reported online social data is subject to various biases @xcite , this data has enabled many applications in the health domain, including tracking the spread of influenza @xcite , detecting the reports of the novel coronavirus @xcite , and identifying various illness reports @xcite . one of the well-studied areas in online public health monitoring is the extraction of adverse drug reactions (adr) from social media data. adrs are the unintended effects of drugs for prevention, diagnosis, or treatment. the researchers in @xcite reported that consumers, on average, report the negative effect of drugs on social media 11 months earlier than other platforms. this highlights the importance of this task. another team of researchers in @xcite reviewed more than 50 studies and reported that the prevalence of adrs across multiple platforms ranges between 0.2% and 8.0%, which justifies the difficulty of this task. in fact, despite the long history of this task in the research community @xcite , for various reasons, the performance of the state-of-the-art models is still unsatisfactory. social media documents are typically short and their language is informal @xcite . additionally, the imbalanced class distributions in adr task has exacerbated the problem. in this study we propose a novel model for extracting adrs from twitter data. our model which we call view distillation (vid ) relies on the existence of two views in the tweets that mention drug names. we use unlabeled data to transfer the knowledge from the classifier in each view to the classifier in the other view. additionally, we use a finetuning technique to mitigate the impact of noisy pseudo-labels after the initialization @xcite . as straightforward as it is to implement, our model achieves the state-of-the-art performance in the largest publicly available adr dataset, i.e., smm4h dataset. our contributions are as follows: 1) we propose a novel algorithm to transfer knowledge across models in multi-view settings, 2) we propose a new technique to efficiently exploit unlabeled data in the supervised adr task, 3) we evaluate our model in the largest publicly available adr dataset, and show that it yields an additive improvement to the common practice of language model pretraining in this task. to our knowledge, our work is the first study that reports such an achievement. next, we provide a brief overview of the related studies. ## related work researchers have extensively explored the applications of ml and nlp models in extracting adrs from user-generated data. perhaps one of the early reports in this regard is published in @xcite , where the authors utilize the related lexicons and extraction patterns to identify adrs in user reviews. with the surge of neural networks in text processing, subsequently, the traditional models were aggregated with these techniques to achieve better generalization @xcite . the recent methods for extracting adrs entirely rely on neural network models, particularly on multi-layer transformers @xcite . in the shared task of smm4h 2019 (weissenbacher and gonzalez-hernandez, 2019), the top performing run was bert model @xcite pretrained on drug related tweets. remarkably, one year later in the shared task of smm4h 2020 @xcite , again a variant of pretrained bert achieved the best performance @xcite . here, we propose an algorithm to improve on pretrained bert in this task. our model relies on multi-view learning and exploits unlabeled data. to our knowledge, our model is the first approach that improves on the domain-specific pretrained bert. ## proposed method our model for extracting the reports of adverse drug effects rely on the properties of contextual neural word embeddings. previous research on word sense disambiguation (wsd) @xcite has demonstrated that contextual word embeddings can effectively encode the context in which words are used. although the representations of the words in a sentence are assumed to be distinct, they still possess shared characteristics. this is justified by the observation that the techniques such as self-attention @xcite , which a category of contextual word embeddings employ @xcite , rely on the interconnected relations between word representations. this property is particularly appealing when documents are short, therefore, word representations, if are adjusted accordingly, can be exploited to extract multiple representations for a single document. in fact, previous studies have demonstrated that word contexts can be used to process short documents, e.g., see the models proposed in liao and grishman (2011) and @xcite for event extraction using hand-crafted features and contextual word embeddings respectively. therefore, we use the word representations of drug mentions in user postings as the secondary view along the document representations of user postings in our model. given the two views we can either concatenate the two sets of features and train a classifier on the resulting feature vector or use a co-training framework as described in @xcite . however, the former is not exploiting the abundant amount of unlabeled data, and the latter is resource intensive, because it is iterative, and also it has shown to be effective only in semi-supervised settings where there are only a few hundred training examples available. therefore, below we propose an approach to effectively use the two views along the available unlabeled data in a supervised setting. in the first step, we assume the classifier in each view is a student model and train this classifier using the pseudo-labels generated by the counterpart classifier. since the labeled documents are already annotated, we carry out this step using the unlabeled documents. more concretely, let l and u be the sets of labeled and unlabeled user postings respectively. moreover, let l d and l g be the sets of representations extracted from the document and drug views of the training examples in the set l; and let u d and u g be the document and drug representations of the training examples in the set u . to carry out this step, we train a classifier c d on the representations in l d and probabilistically, with temperature t in the softmax layer, label the representations in u d . then we use the association between the representations in u d and u g to construct a pseudo-labeled dataset of u g . this dataset along its set of probabilistic pseudo-labels is used in a distillation technique @xcite to train a classifier called c g . correspondingly, we use the set l g to train a classifier c g , then label the set u g and use the association between the data points in u g and u d to construct a pseudo-labeled dataset in the document view to train the classifier c d . the procedure above results in two classifiers c d and c g . the classifier in each view is initialized by the knowledge transferred from the other view. however, the pseudo-labels that are used to train each classifier can be noisy. thus, in order to reduce the negative impact of this noise, in the next step, we use the training examples in the sets l d and l g to further finetune these two classifiers respectively. to finetune c d we use the objective function below: algorithm 1 illustrates our model (vid) in structured english. on lines 8 and 9 we derive the document and drug representations from the sets l and u . on lines 10 and 11 we use the labeled training examples in the two views to train c d and c g . on algorithm 1 overview of vid 1: procedure vid 2: given: 3: l : set of labeled documents 4: u : set of unlabeled documents 5: return: 6: two classifiers c d and cg 7: execute: 8: derive two sets of representations l d and lg from l 9: derive two sets of representations u d and ug from u 10: use l d to train classifier c d 11: use lg to train classifier cg 12: use c d to probabilistically label u d 13: transfer labels of u d to ug and use them to train cg 14: finetune cg using equation 2 15: use cg to probabilistically label ug 16: transfer labels of ug to u d and use them to train c d 17: finetune c d using equation 1 18: return c d and cg lines 12-14 we train and finetune c g , and on lines 15-17 we train and finetune c d . finally, we return c d and c g . in the next section, we describe our experimental setup. ## experimental setup we evaluated our model in the largest publicly available adr dataset, i.e., the smm4h dataset. this dataset consists of 30,174 tweets. the training set in this dataset consists of 25,616 tweets of which 9.2% are positive. the labels of the test set are not publicly available. the evaluation in the dataset must be done via the codalab website. we compare our model with two sets of baselines: 1) a set of baselines that we implemented, 2) the set of baselines that are available on the codalab website foot_2 . our own baseline models are: bert, the base variant of the pretrained bert model @xcite , as published by google. bert-d, a domain-specific pretrained bert model. this model is similar to the previous baseline, however, it is further pretrained on 800k unlabeled drug-related tweets that we collected from twitter. we pretrained this model for 6 epochs using the next sentence prediction and the masked language model tasks. bert-d-bl, a bi-directional lstm model. in this model we used bert-d followed by a bi-directional lstm network @xcite . we also compare our model with all the baselines available on the codalab webpage. these baselines include published and unpublished models. they also cover models that purely rely on machine learning models and those that heavily employ medical resources; see @xcite for the summary of a subset of these models. we used the pytorch implementation of bert @xcite . we used two instances of bert-d as the classifiers in our model-see figure 1 . please note that using domain-specific pretrained bert in our framework makes any improvement very difficult, because the improvement in the performance should be additive. we used the training set of the dataset to tune for our two hyperparameters t and λ. the optimal values of these two hyper-parameters are 2 and 0.5 respectively. we trained all the models for 5 epochs foot_3 . during the tuning, we observed that the finetuning stage in our model requires much fewer training steps, therefore, we finetuned for only 1 epoch. in our model, we used the same set of unlabeled tweets that we used to pretrain bert-d. this verifies that, indeed, our model extracts new information that cannot be extracted using the regular language model pretraining. as required by smm4h we tuned for f1 measure. in the next section, we report the f1, precision, and recall metrics. ## results and analysis in summary, we evaluated our model in the largest publicly available adr dataset and compared with the state-of-the-art baseline models that use domain specific language model pretraining. we showed that our model outperforms these models, even though it uses a smaller unlabeled corpus. we also carried out a set of experiments and demonstrated the efficacy of our proposed techniques. ## conclusions in this study we proposed a novel model for extracting adverse drug effects from user generated content. our model relies on unlabeled data and a novel technique called view distillation. we evaluated our model in the largest publicly available adr dataset, and showed that it outperforms the existing bert-based models.