Sentence Contextual Encoder with BERT and B i LSTM for Automatic Classification with Imbalanced Medication Tweets (2020)

Abstract: This paper details the system description and approach used by our team for the SMM4H 2020 competition, Task 1. Task 1 targets the automatic classification of tweets that mention medication. We adapted the standard BERT pretrain-then-fine-tune approach to include an intermediate training stage with a biLSTM architecture neural network acting as a further fine-tuning stage. We were inspired by the effectiveness of within-task further pre-training and sentence encoders. We show that this approach works well for a highly imbalanced dataset. In this case, the positive class is only 0.2% of the entire dataset. Our model performed better in both F1 and precision scores compared to the mean score for all participants in the competition and had a competitive recall score.

## introduction social media is ubiquitous, a continuous part of our daily lives; it offers new ways of communication and contains important data for various disciplines @xcite . this is especially crucial for health related sectors. twitter posts are now recognised as an important source of patient-generated data, providing unique insights into population health @xcite . however, tweets contain a lot of noise and are imbalanced in their nature. a fundamental step towards incorporating twitter data in pharmacoepidemiologic research is to automatically recognise medication mentions in tweets. hence task 1 in smm4h 2020 (ari z. klein and gonzalez-hernandez, 2020) is proposed based on a similar task from 2018 @xcite to address these challenges. the task is aimed at identifying if tweets contain information about medications using a highly skewed, imbalanced training dataset in which only 0.2% contains a positive label. ## conclusion the results of our experiments suggest that extracting features from language models and using them for intermediate modelling can provide a good platform to study the automatic classification of medical tweets, even if the final target dataset is highly imbalanced. moreover, interestingly, various layers of features from language models can be used as a sequence in the representation, and this further confirms that different layers of representation in complex language models, such as bert, contain various levels of information semantics.