BERT Implementation for Detecting Adverse Drug Effects Mentions in R ussian (2020)

Abstract: This paper describes a system developed for the Social Media Mining for Health 2020 shared task. Our team participated in the second subtask for Russian language creating a system to detect adverse drug reaction presence in a text. For our submission, we exploited an ensemble model architecture, combining BERT’s extension for Russian language, Logistic Regression and domain-specific preprocessing pipeline. Our system was ranked first among others, achieving F-score of 0.51.

## introduction in this paper, we focus on the problem of discovering the presence of adverse drug reaction (adr) concepts in twitter posts as part of the the social media mining for health applications (smm4h) shared task @xcite . the paper is based on the participation of our team in the russian language segment of the second task: adr presence classification. organizers of smm4h 2020 task 2 provided datasets of russian tweets with binary annotation indicating the presence or absence of adrs in each post. the aim of the task was to develop a system to classify the tweets according to the presence of adrs. texts were given in a raw form, so they contained misspellings, slang, emojis, hashtags, usernames and were quite noisy. this year is the first time for a distinct set of russian tweets to be included in the task. we tested and compared several different approaches for solving such type of classification task, including classical machine learning approaches and neural networks, and also different preprocessing pipelines. ## conclusion in this work, we have explored an application of bidirectional encoder representations from transformers (bert) to the task of text classification in russian. we have empirically evaluated different versions of tuned rubert and preprocessing pipelines against f-score for the "positive" class and experiments have shown that logistic regression trained on the result of a six conversational rubert models ensemble trained on the undersampled data with light preprocessing and tokenized punctuation outperforms every other model, providing a new baseline for adr presence classification in russian with f-score 0.51, precision 0.45 and recall 0.60 on the test data.