Medication Mention Detection in Tweets Using ELECTRA Transformers and Decision Trees (2020)

Abstract: This study describes our proposed model design for the SMM4H 2020 Task 1. We fine-tune ELECTRA transformers using our trained SVM filter for data augmentation, along with decision trees to detect medication mentions in tweets. Our best F1-score of 0.7578 exceeded the mean score 0.6646 of all 15 submitting teams.

## introduction the social media mining for health applications (smm4h) shared task involves natural language processing challenges for using social media data for health research. we participated in the smm4h 2020 task 1, focusing on automatic classification of tweets that mention medications @xcite . this binary classification task involves distinguishing tweets that mention a medication or dietary supplement (annotated as '1') from those that do not (annotated as '0'). this task was first organized in 2018 using a data set containing an artificially balanced distribution of the two classes @xcite . several approaches have been presented to address this binary classification task @xcite @xcite . however, this year's task is more challenging. the data set represents a natural, highly imbalanced distribution of the two classes from tweets posted by 112 women during pregnancy, with only approximately 0.2% of the tweets mentioning a medication @xcite . this paper describes the ncuee (national central university, dept. of electrical engineering) system for the smm4h 2020 task 1. to deal with highly imbalanced distribution, the support vector machine trained using the training data is used as a filter to crawl and select more tweets for data augmentation. we then fine-tune the pre-trained electra transformers @xcite , using our augmented data for medication mention detection. in addition, we train the decision tree as a supplementary classifier. finally, the integrated set of testing instances are detected as a positive class from electra and decision trees are regraded as label '1' and the remaining cases as label '0' to form our submissions. ## the ncuee system in addition to the training data provided by task organizers, we crawl and select highly related tweets for data augmentation. we manually check small positive tweets to pick up textual terms that may refer to medications, and then use these terms as seeds for query expansions. the pre-trained word2vec embedding from twitter data is used to look up word vectors and compare their cosine similarities. the top 10 similar terms of seeds are collected, where expanded terms are kept if the document frequency (df) of an expanded term in the positive class exceeds that in the negative class. each expanded term, along with the query term 'pregnant' is regarded as an individual query to search for possibly related tweets from twitter. to automatically label highly positive cases, we train the support vector machine (svm) using the provided training data and select crawled tweets predicted to be positive cases by the svm. finally, we construct an augmented data set including the original training sets for neural computing. electra (efficiently learning as encoder that classifiers token replacements accurately) is a new pre-training approach that aims to match or exceed the downstream performance of an mlm (masked language modeling) pre-trained model while using less computational loading @xcite . electra trains two transformer models: the generator, which replaces the tokens in a sequence for training a masked language model; and the discriminator, which tries to identify which tokens in the sequence were replaced by the generator. we use pre-trained electra transformers and fine-tune them using our augmented data to detect medication mentions in tweets. according to our empirical results from the validation set, the decision tree (dt) classifiers usually achieved a high degree of precision if the discriminated features had been extracted and learned, but very low recall if the testing cases were significantly different from the trained ones. hence, we use the same trained svm as a filter to select the positive cases (predicted as '1' ) of from the 2018 task training data that may be closely similar to the positive tweets in this task and include these in an augmented set. we then adopt the tf-idf (term frequency-inverse document frequency) weighting method to extract discriminated features of positive tweets from this augmented set and use them with the original training data to train the decision trees. finally, based on our error analysis, we found the decision tree classifier was partially complementary to electra. so, the integrated set of testing instances predicted as the positive class from the elec-tra transformers and decision trees are labeled '1', otherwise '0 ' in our submissions. ## evaluation we picked up 112 seeds from 181 positive tweets to further expand the dataset by 72 unique terms via cosine similarity through the pre-trained word2vec embeddings of the tweets. without using the svm as a filter, we have 57,678 positive tweets and 105,273 negative tweets. with svm, we have 32,619 positive tweets and 65,238 negative tweets. the distribution of tweets after data augmentation (da) was still remained imbalanced, with a positive to negative ratio close to 1:2. the pre-trained electra-large was downloaded from huggingface @xcite . the hyper-parameters used for fine-tuning electra are as follows: batch size 16; gradient accumulation steps 16; learning rate 1e-5; and number of training epochs 6. ## conclusions this study describes the ncuee system participating in the smm4h 2020 task 1 for medication mention detection, including system design, implementation and evaluation. our best f1-score of 0.7578 exceeded the mean score 0.6646 for all 15 teams with at least one submissions.