Sentence Transformers and B ayesian Optimization for Adverse Drug Effect Detection from T witter (2020)

Abstract: This paper describes our approach for detecting adverse drug effect mentions on Twitter as part of the Social Media Mining for Health Applications (SMM4H) 2020, Shared Task 2. Our approach utilizes multilingual sentence embeddings (sentence-BERT) for representing tweets and Bayesian hyperparameter optimization of sample weighting parameter for counterbalancing high class imbalance.

## introduction and related work automatic adverse drug reaction detection from social media has high significance to health informatics and pharmacovigilance due to fast, scalable, and diverse public health surveillance opportunities. numerous studies have proposed natural language processing and machine learning solutions to detect mentions of adverse drug effects, especially from twitter @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . proposed approaches in earlier studies vary from rule-based systems to deep learning. the challenge of detecting adverse drug effect mentions in twitter remains unsolved due to lack of large-scale annotated datasets, rareness of relevant tweets among all tweets as well as ever-changing nature of the phenomenon. the task consists of building separate adverse drug effect mention detection (binary classification) models for english, french, and russian datasets from twitter (9.25%, 1.61%, and 8.75% positive class prevalence, respectively). detailed description of data and task can be found in @xcite . ## discussion and conclusions we chose to use pre-computed sentence embeddings instead of fine-tuning on token embeddings out of original bert models due to its low computational requirements @xcite . while it is possible to represent tweets with a standard bert model as well (e.g. by averaging the token embeddings out of the output layer), such sentence representations were shown to be inferior to embeddings specifically trained for representing sentences @xcite . bayesian optimization is beneficial in settings where the function to be minimized/maximized is a black-box function without a known closed-form, expensive to evaluate, and stochastic @xcite . as f (θ) corresponds to cross-validation performance in our case, it is indeed a black-box function and computationally expensive to evaluate. furthermore, it is a stochastic function as the same θ may give different outputs for different runs due to randomness in neural network weight initialization. that is our motive for employing bayesian hyperparameter optimization for sample weighting. advantage of using a gaussian process as the surrogate model is that it can provide uncertainty estimations. as expected, optimum positive sample weight parameters found by bayesian optimization are different for each dataset and french dataset requires much higher balancing due to its extreme class inbalance. our bayesian optimization approach can be easily extended to optimization of other hyperparameters of the model including neural network architecture and other relevant design choices.