Leveraging knowledge sources for detecting self-reports of particular health issues on social media (2021)

Abstract: This paper investigates incorporating quality knowledge sources developed by experts for the medical domain as well as syntactic information for classification of tweets into four different health oriented categories. We claim that resources such as the MeSH hierarchy and currently available parse information are effective extensions of moderately sized training datasets for various fine-grained tweet classification tasks of self-reported health issues.

## introduction social media are a ubiquously accessible way to communicate and interact with others, making their users producers of big data at a fast rate. it is estimated that about 500m tweets are sent each day on twitter which often contain information about opinions, trends, reviews, health, incidents, etc. this offers the possibility to gain insight into individuals' behavior and general state in direct and unmitigated fashion @xcite . health applications based on social media are an active research area for outbreak management, disease surveillance @xcite , and pharmacovigilance @xcite . for instance, epidemiologists hope to mine social media to predict and monitor the likelihood and possible severity of outbreaks in a timely fashion. systems that support this type of research have to make predictions from incomplete data of varying quality. deep learning methods are popular for nlp applications and demonstrated significant improvements in areas such as text classification. deep models have been widely used for personal health mention detection @xcite , @xcite , @xcite , @xcite , @xcite . deep models, however, do not usually have access to outside resources, apart from word embeddings. while such models can outperform systems that are limited to look-up in gazetteer lists for task specific terms, this can only be when the terms of the test set are foreshadowed sufficiently in the training set. sensitivity to lexical triggers is crucial in classification, especially in the medical domain, where vocabularies are ever-growing and new specialized terms are introduced everyday. the most recent example is the term "covid" which was coined in late 2019. most language models are trained contextually. the assumption for contextualized language models is that the meaning of a word can be represented by the context in which it appears. however, the context usually is not sufficient to represent the meaning for rare specialized terms, which require large amounts of training data for coverage. in addition, highly specialized terms with very different meanings may occur in the same immediate context (see example 1), rendering contextualized word embeddings less effective. the context for hydrocephalus and leukemia here is the same, and is the same for all diseases, making such contextualized word embeddings less sensitive to, for instance, the more fine-grained distinctions between birth defects and cancer. consequently, contextualized language models often fail to make these distinctions. current language models have over 60m parameters foot_0 , making fine-tuning as well as testing timeconsuming and requiring large training sets. these issues motivate us to investigate widely available external knowledge sources, such as mesh @xcite , and language features in a deep architecture suitable for personal health mention detection. we show that knowledge sources, combined with light-weight word embeddings and language models such as glove @xcite and elmo @xcite , are strong contenders for larger models such as roberta @xcite . we experiment on four health-related tweet classification tasks of the ongoing smm4h workshop and present ablation studies to assess the contribution of different external knowledge sources. our results suggest that the external resources tested indeed enhance performance when properly calibrated to work together. best performance is achieved with a two layer system that adds representations of gazetteer lists and enhanced partof-speech annotations in an encoder followed by a graph convolutional neural network (gcnn) (kipf and welling, 2017) representing preprocessed grammatical dependencies. ## related literature personal experiences posted on social media can give insight into the state of public health. examination and identification of smoking behavior @xcite , non-medical use of opioids @xcite , and identification of medicationrelated experiences @xcite , @xcite have recently been studied on social media. a variety of models have been proposed for the task addressed in the current paper, namely health experience mention detection for different experiences. the approaches fall into three main categories, namely statistical models with hand-crafted features, pure deep learning models, and deep models with leveraged features. hand-crafted features @xcite proposed a set of textual features such as count of emotion words, of unique words, of first person pronouns, of pronouns, etc., for personal health surveillance. @xcite compared different word embeddings such as glove, word2vec @xcite , fasttext, and wordrank with the features of @xcite . their word embeddings performed close to one another and considerably outperform their feature based model. for detection of vaccination behaviour, @xcite proposed to use the count of pos tags, number of special characters (such as # and @), and count of emotion words as input features to an ensemble of svm, logistic regression and random forest classifiers. @xcite also experimented with a pure deep-learning method (employing ulmfit @xcite with finetuning) and reported a performance close to their feature-based model, demonstrating that a model with handcrafted features is a strong contender for deep models. to identify drug and adverse drug reaction mentions, @xcite used a svm classifier with some hand-crafted features such as the count of typed-dependency relation, drug names, and sentiment score and demonstrated success to some extent. (c ¸öltekin and rama, 2018) also addressed the task using a svm model with word and character n-grams as input features. bag of word features (tf-idf) as well as negation, adverse reaction mentions, and drug mention were also used in @xcite as input for a svm. models with hand-crafted features have shown competitive performance compared to deep models on some tasks and datasets but despite the availability of many high quality resources from which features can be derived, the power of contextualized word embeddings to fill in not only lexical gaps, but also subtask specific patterns led to the investigation of deep models. pure deep models a majority of the proposed models for personal health detection are deep learning based. studies such as @xcite and @xcite use conventional pre-trained word embeddings, such as word2vec @xcite and glove. by the advent of pre-trained language models, many studies benefit from models such as ulmfit @xcite , elmo @xcite , and bert @xcite , @xcite , @xcite , (aroyehun and gelbukh, 2019), (babu and eswari, 2020), @xcite . deep models are dependent on the representativeness of their training data for the test cases. pretrained (often bert-based models) are generally the top performers in current shared task competitions. the models have to be developed by highly skilled machine learning experts. the data, on the other hand, have to be collected and annotated by domain experts, if they are to be of use in health care research. deep models with leveraged features while many approaches to health-oriented classification of tweets use features in statistical models, there have been fewer efforts to leverage them in deep models. for the task of adverse drug reaction mention detection @xcite proposed to embed pos tags, gazetteer information, and sentiment scores, then concatenate the features to glove embeddings as input to a hybrid of cnn and lstm. pos tags as well as features such as side effects, medical concepts, and first character are concatenated to word embeddings in @xcite . @xcite also leveraged features such as adverse mentions, pos tags, and scope of negation and modality, by concatenation to word2vec and glove embeddings. the limitation of input feature concatenation is that only a fixed number of annotation types can be used. adding more gazetteer lists requires reconfiguring the network, since the number of dimensions is bounded by the number of predefined annotations. this is undesirable because addition of any annotation type has to be performed by a machine learning expert and must be followed by re-training the network from scratch. in the following we outline a simple architecture, where the end-user (possibly an epidemiologist) can add or remove gazetteer lists and fine-tune the model without making any changes in the network settings such as hidden dimensions (and thus without requiring help from a machine learning expert). ## tasks in order to demonstrate the ability of the presented approach to adapt to new domains, we compare performance on four tasks from the social media mining for health application foot_2 (smm4h) shared tasks. all tasks involve detection of self-reported health mentions on twitter. sm18-2: self-reported medication intake is a 3class. tweets which clearly express personal medication intake are considered category 1. tweets where the user may have taken some medication are category 2. category 3 tweets mention medication names but do not indicate personal intake @xcite . the performance is evaluated as µf score of class 1 and class 2. foot_3 sm18-4 vaccination behavior mention classification is a binary task where the positive class indicates the user has received or intends to receive a flu vaccine. a tweet is classified as negative if it does not contain any mentions of a vaccination or if it merely mentions vaccination @xcite . sm19-1 adverse drug reaction mention is the task of identifying mentions of side effects as the results of drug consumption @xcite . ## external resources we experiment with two types of external knowledge sources for the deep learning system: (a) gazetteer lists extracted from mesh as examples of a high quality resource developed by experts that can be used to partly define the domain of the task and (b) language features (pos, nes, dependencies) extracted from the text with a parser and named entity recognition pipeline. ## proposed model we developed a multi-layer system which includes four layers, namely: embeddings, self-attention, gcnn, and classification. layer1: embeddings we combine traditional word embeddings with pos embeddings and our gazetteer embeddings additively. our approach is to apply word2vec on pos tags instead of tokens. the embeddings are trained using the gensim package @xcite with a window size of @xmath0. the pretraining is performed on training data of all task introduced in section 3. the resulting embeddings are used to initialize a pos embedding matrix @xmath1, where φ is the number of distinct pos tags, and d emb is the dimensionality of the word embeddings. the pos embeddings are fine-tuned during the training for the main classification task. the vectors e hydrocephalus , g birthdef , and p n n p are added to form h 1 3 , the aggregate for time-step 3 in layer 1. the additive approach enables the model to encode as many as gazetteer annotations, without introducing new dimensions to the token representations (in contrast to concatenative approaches). after the training, one can easily introduce a new gazetteer annotation y by adding a learnable vector g y to the model parameters, and only fine-tune the model, without making any changes to hidden dimensions. layer2: self-attention encoder we use a selfattention encoder (the encoder part of the transformer) as first layer @xcite . the encoder at layer 2 gets the representations h 1 i and outputs representations h 2 i . the number of heads in the multi-head attention is n @xmath2. layer3: graph cnn we use a graph convolutional network (gcnn) @xcite to encode the dependency graph following @xcite . in gccn, each token is represented based on its adjacent tokens in a dependency parse using: the normalized scores are then used for a weighted sum @xmath3. the final vector h is used as input to a linear layer for the classification. the proposed model is implemented using the pytorch library @xcite . crossentropy is used to calculate the network loss and the model is optimized using the adam optimizer @xcite . ## numerical results we evaluate the proposed model using a set of ablation studies. the sm19-1 and sm20-5 tasks are evaluated on the official test data. for sm18-2 and sm18-4, official test data is not available, therefore we replicate the state-of-the art systems and perform evaluation on a hold-out set from the original training data. the gazetteer lists and named-entity features provide considerable improvements for all tasks except for sm18-4 with marginal improvements. note that sm18-4 is the vaccination behaviour prediction task, specific to flu. this result is to be expected: it requires identifying self reports, but the trigger terms for the flu domain consisted only of flu, making gazetteers ineffective. the tasks with more diverse vocabularies show greater impact of gazetteer lists. combining grammatical and gazetteer features robustly yields best results. interestingly, adding knowledge resources to a lighter language model approaches performance of a larger model. for instance, glove with all resources outperforms elmo without resources for all tasks and even approaches roberta or biobert without resources. we also observe that while our system configurations using roberta or biobert reported in ## case-study to examine the effects of knowledge sources, we probe the attention importance scores at layer4 (equation 2 ). the scores demonstrate how the the model attends to different tokens. we probe the scores in two cases, with and without gazetteer lists. figure 3 and figure 4 demonstrate visualizations of the attention scores for two samples from sm20-5 and sm19-1 tasks respectively. higher attention scores are indicated with darker gray color. the model of figure 3a uses no gazetteer lists. the model partially attends to the birth defect mention trisomy18 and pays no attention to the pregnancy issue stillbirth. the model, however, properly attends to the personal pronouns i and lexical triggers such as baby and birth. on the other hand, when the model is given birthdef and pregnancy gazetteers, the model puts more attention on still-birth as evidence for a birth defect, leading to a more certain prediction. a similar pattern is observed in figure 4 . supply-this paper demonstrates the effectiveness of using gazetteer lists from high-quality sources, standard named entity categories and part-of-speech embeddings with a self-attention encoder and a gcnn encoding grammatical dependencies. the architecture supports precision oriented domain adaptation from widely available, high-quality resources (i.e. mesh). adaptation with new gazetteer lists using additive annotation sidesteps the need to reconfigure or retrain the neural networks the experiments confirm that quality external resources can offset the lower parameter space of light-weight word embedding/language models, such as glove and elmo. at the same time, these resources effectively combine with roberta for best performance. the stronger improvements in precision are especially promising for health applications. the comparative results on different tasks and different domains proves that this extensible architecture is well-suited for actual use in the wild on domains and tasks, for which experts know to supply high-quality terminology resources. after taking olanzapine i wake up and feel like i am in a straight jacket because my muscles feel stiff after taking olanzapine i wake up and feel like i am in a straight jacket because my muscles feel stiff a) without drug and anatomy gazetteers b) with drug and anatomy gazetteers figure 4: visualization of attention scores for a sample from sm19-1