Sentence Classification with Imbalanced Data for Health Applications (2020)

Abstract: Identifying and extracting reports of medications, their abuse or adverse effects from social media is a challenging task. In social media, relevant reports are very infrequent, causes imbalanced class distribution for machine learning algorithms. Learning algorithms typically designed to optimize the overall accuracy without considering the relative distribution of each class. Thus, imbalanced class distribution is problematic as learning algorithms have low predictive accuracy for the infrequent class. Moreover, social media represents natural linguistic variation in creative language expressions. In this paper, we have used a combination of data balancing and neural language representation techniques to address the challenges. Specifically, we participated the shared tasks 1, 2 (all languages), 4, and 3 (only the span detection, no normalization was attempted) in Social Media Mining for Health applications (SMM4H) 2020 (Klein et al., 2020). The results show that with the proposed methodology recall scores are better than the precision scores for the shared tasks. The recall score is also better compared to the mean score of the total submissions. However, the F1-score is worse than the mean score except for task 2 (French).

## introduction advances in representation learning that attempts to automatically learn features for natural language processing @xcite present the possibility of utilizing social media (i.e. twitter) data source for public health applications such as health monitoring and surveillance. several ethical, legal and methodological challenges need to be addressed that are unique to twitter data source @xcite . the ongoing shared tasks in social media mining for health applications (smm4h) define evolving challenges specific to the twitter data source for health domain @xcite . to address the methodological challenges, in recent years, several techniques have been proposed based on the smm4h shared tasks @xcite . epidemiologists intend to detect mentions of health issues related to medications early on twitter. the adverse effect of medications is one of the leading causes of post-therapeutic deaths @xcite . one of the challenges of detecting real reports on medications, their abuse or adverse effects is to distinguish the relevant true reports from other general statements, news, and institutional advice. reports on health issues (i.e. abuse or adverse effects of a medication) in social media are rare instances (i.e. very small percentage contain relevant information) @xcite . this causes a major machine learning methodological challenges called imbalanced learning problem @xcite . the learning problem happens in the presence of underrepresented data and severely skewed class distribution, the situation where one of the class categories comprises a significantly larger proportion of the dataset than the other classes. imbalanced class distribution is a practical issue in most real-world datasets (e.g. fraud detection, disease detection) and complicates learning when the identification of the minority class is of specific importance. this is a general problem for health domain with medical data utilizing machine learning methods @xcite . ## problem statement and data description in this paper, six datasets from the social media mining for health application (smm4h) 2020 shared tasks were used for health monitoring and surveillance challenges. all the shared task datasets are labeled and can be modeled in one of three popular types of supervised classification problems. binary (b) classification is the task of classifying the elements of a given set into two groups whereas multiclass (m) classification generalizes into more than two groups. span detection (d) and normalization (n) task is defined as d+n which can be modeled as multi-class classification task followed by named-entity recognition modeling @xcite . task 2 is another binary classification task that involves classifying tweets based on the mentions of adverse effect (ae) of a medication. this task includes distinct sets of tweets posted in three languages: english (en), french (fr) and russian (ru). in tab. 1, rows corresponding to task id t2 (en), t2 (fr) and t2 (ru) refer to sub-tasks in three languages. tweets that mention an adverse effect of medications are labeled as "1" and those that do not have mention are labeled as "0". the tasks require taking into account subtle linguistic and semantic variations between aes and indications (i.e. the reason for using the medication). the t2 (fr), t2 (en) and t2 (ru) datasets represent moderately imbalanced class distribution with ir of 1.61%, 9.25% and 8.75% respectively. task 3 involves detecting the span of tweet containing an adverse effect (ae) of a medication and then mapping the extracted ae to a standard concept identifier (id) in the meddra vocabulary (preferred terms). the training data includes tweets that report or indicate an ae that are labeled as "1" and those that do not mention are labeled as "0". the detection task thus requires a model to distinguish between aes and indications. the class distribution of the dataset is balanced (@xmath0.20%) for the detection task. the normalization task involves classification to multiple classes where each class can be defined as a standard meddra id. we have not attempted the normalization part of the task and left for future work. task 4 requires distinguishing between more than two classes of tweets and can be modeled as a multi-class classification problem. the task involves distinguishing among tweets that mention at least one prescription opioid, benzodiazepine, atypical anti-psychotic, central nervous system stimulant or gaba analogue. tweets that report potential abuse/misuse are labeled as "a" from those that report non-abuse/-misuse consumption which are labeled as "c", merely mention of the medication are labeled as "m", and unrelated are labeled as "u". the task has moderately imbalanced class distribution with ir of 15.99%. ## methodology for classification tasks we have developed systems for shared tasks 1, 2 (all languages), 4, and 3 (only the span detection, no normalization was attempted). the main focus was towards the imbalanced classification tasks. in this section, we will describe the methods used for five classification tasks and in the next section, we will describe the method developed for the span-detection part of task 3. the models were trained on the training dataset and evaluated on validation dataset; test data was not available when the methods were being developed. each solution of the binary and multi-class classification task can be subdivided into four common steps and these steps will be elaborated in the following subsections. ## methodology for span detection task task 3 can be divided into two sub-tasks: span detection and concept normalization. we have worked on the detection part of the task and the normalization part is left for the future work. the dictionarybased simple traditional approach has been used in information detection task @xcite . the approach utilizes a carefully constructed dictionary to identify and tag the related entities from the tweets. for task 3, we have used a dictionary-based search approach to detect the span of adverse effects. a dictionary was generated from the amia shared task foot_4 with 6,649 annotated instances foot_5 of adverse effects. ## results and analysis the official performance evaluation metrics for the tasks are precision (p), recall (r), and f1-score (f1) computed on the positive class (i.e. minority class). using the methodology described in this paper, from the table we can observe that the recall score is higher compared to the precision score in most cases. as positive labels refer to the minority class, there are a larger number of negative examples that could become false positives (fp). conversely, there are fewer positive examples that could become false negatives (fn). the disproportion in class distribution 143 can result in the disproportion of the fps and fns which could result in high recall and low precision score. as f1-score is a weighted combination of these two matrices, a low precision score can cause an overall low f1-score. although random under-sampling was used to balance the training dataset, it does not improve the precision score on the validation and test dataset since the validation and test dataset are still imbalanced that reflect the real class distribution. ## conclusion this paper reports the preliminary exploration of twitter-based imbalanced data modeling techniques for health applications. we have explored techniques for efficient syntactic and semantic representation of tweets. as the focus was on imbalanced class distribution, random under-sampling with novel noise reduction technique was utilized to balance the dataset. the traditional margin-based and ensemble tree-based classifiers were used to classify the tweets. the directions for future works involve deeper exploration and extensive methodological improvements to enhance the learning performance. for example, the statistical learning theorems establish the number of examples needed to estimate the accuracy of a classifier as a function of its complexity (vcdimension). however, the class imbalance does not enter these formulas anywhere @xcite . overall, a detailed exploration of classification models and evaluation metrics can be done to enhance model performance for skewed data distribution.