NLP @ VCU : Identifying Adverse Effects in E nglish Tweets for Unbalanced Data (2020)

Abstract: This paper describes our participation in the Social Media Mining for Health Application (SMM4H 2020) Challenge Track 2 for identifying tweets containing Adverse Effects (AEs). Our system uses Convolutional Neural Networks. We explore downsampling, oversampling, and adjusting the class weights to account for the imbalanced nature of the dataset. Our results showed downsampling outperformed oversampling and adjusting the class weights on the test set however all three obtained similar results on the development set.

## introduction this paper describes our participation in task 2 of the social media mining for health application (smm4h) 2020 challenge to automatically identify adverse effects (ae) in english tweets. to address this challenge, we explored a supervised binary classification system to automatically identify the aes using convolutional neural networks (cnns). in order to deal with the unbalanced nature of the data set, we explored downsampling, oversampling, and utilization of the class weights. ## methods in this section, we discuss our ae identification system. our system can be found here 1 . feature representation. we pre-process the data as follows: 1) the html symbol &amp; are removed as per @xcite ; 2) double-quotes are removed; 3) hashtags, links, and usernames are replaced with the string "hashtag", "link" and "username" respectively as per @xcite ; 4) emojis are substituted with a phrase that represents that emoji as per @xcite ; 5) tweets are lowercased. each word in the tweet is represented as an embedding. we evaluated three embedding types, glove @xcite , word2vec @xcite and fasttext @xcite , trained over different corpora types. our resulting system described here uses glove trained on twitter. algorithm. we evaluated using convolutional neural network (cnns) as per @xcite . one of the beneficial properties of cnn is that it preserves the spatial orientation, in this case, the sequence of the words in the tweet. cnns consist of four layers: 1) embedding layer -to encode words in sentences by real-valued vectors; 2) convolution layer -to get local features from each part of the input; 3) pooling layer -to extract the most relevant features, and 4) feed-forward layer -a fully connected layer to perform classification. for this, we first feed each tweet into cnn to learn the ae representation of the tweet. second, we apply the convolution layer to learn the local features from the embedding vectors obtained from each word of a tweet. next, we apply the max-pooling layer to extract the most important feature. next, we unstack the volume into a flat vector and feed it into the fully connected feedforward layer. finally, the fixed-length vector is fed into a softmax layer to perform the classification. for training, the classification error is back-propagated and the model is re-trained until the error is minimized. the weights of the matrix and bias are the parameters that get tuned until the optimized model is obtained. ## data the training set contains 20,544 tweets: 1,903 tweets contain aes and 18,641 tweets do not contain aes. the development set contains 5,134 tweets: 4,660 tweets are negative and 474 are positive. the training and development data sets are both highly imbalanced containing a 1:10 ratio of ae tweets to non-ae tweets. the test set contains 4,759 tweets. ## results in this section, we report the precision (p), recall (r), and f-measure (f) of our system on the ssm4h task 2 data set for the three runs described above. ## conclusions and future work our cnn model achieved a reported f1 score of 0.35 on the test dataset and 0.50 on the development data. our experimentation with pre-trained word embedding showed that glove trained on twitter is optimal for this task. our results also show that downsampling, oversampling, and keras class weights achieve similar f1 scores with the development data, though downsampling outperformed oversampling and class weights on the test data. in the future, we would like to experiment further in the pre-processing stage. we also plan to explore the possibility of character embeddings or utilizing recurrent neural networks (rnns) in depth. we would like to investigate the usage of additional word embeddings such as bidirectional encoder representations from transformers (bert) for this task as well.