SINAI @ SMM 4 H ’22: Transformers for biomedical social media text mining in S panish (2022)

Abstract: This paper covers participation of the SINAI team in Tasks 5 and 10 of the Social Media Mining for Health (#SSM4H) workshop at COLING-2022. These tasks focus on leveraging Twitter posts written in Spanish for healthcare research. The objective of Task 5 was to classify tweets reporting COVID-19 symptoms, while Task 10 required identifying disease mentions in Twitter posts. The presented systems explore large RoBERTa language models pre-trained on Twitter data in the case of tweet classification task and general-domain data for the disease recognition task. We also present a text pre-processing methodology implemented in both systems and describe an initial weakly-supervised fine-tuning phase alongside with a submission post-processing procedure designed for Task 10. The systems obtained 0.84 F1-score on the Task 5 and 0.77 F1-score on Task 10.

## introduction social media networks are widely used as a base for public health related research. among other websites, twitter constitutes a useful data source for researches due to the real time nature of the content and the ease to accessing publicly available information @xcite . however, extraction of information from tweets remains challenging due to some particularities of the language variety used in the social network. spelling mistakes, shortenings, abbreviations, agrammatical word truncation alongside with frequent use of emoji, hashtags, emoticons and even usage of digits and mathematical symbols to represent phonetic sounds are some characteristics that distinguish linguistic register typically adopted by twitter users @xcite . this fact restricts the efficiency of rule-based information retrieval approaches, such as exact matching, used to extract data in many researches @xcite @xcite . named entity recognition (ner) is a task in natural language processing (nlp) which has as objective structuring free-text data by identifying relevant terms in it. text classification is another way of structuring data which improves information retrieval by assigning standardized labels to texts @xcite . in this paper we present the systems developed by the sinai team for shared tasks 5 and 10 at the social media mining for health (#smm4h) workshop at coling 2022. on the one hand, task 5 brings the community effort to design systems that would automatically classify spanish tweets reporting covid-19 in 3 categories: self reports, non-personal reports and literature/news mentions. the presented tweet classification systems relies on robertiuto @xcite , a language model pre-trained on social media text. on the other hand, the purpose of task 10 (socialdis-ner) is to design systems that would recognize all kinds of disease mentions in tweets written in spanish. with this objective, we developed a system based on a model pre-trained on generaldomain text @xcite . in order to leverage large scale additional silver standard data with automatically generated labels provided by task's organizers we designed a two-stage fine-tuning framework. concerning data processing, we describe a submission post-processing procedure aimed to improve system's performance and we enhanced our systems with additional text pre-processing. ## data the organizers of both tasks made available corpora of tweets in spanish that represent first-hand experience of diseases and other health-related content. ## pre-processing the peculiarities of the linguistic register adopted by twitter users make text pre-processing an essential step in order to get a language model to perform robustly. for both tasks, we followed the same rule-based pre-processing procedure which consists of the following steps: the steps 2-4 were carried out by using the pysentimiento python package @xcite . 4 task 5. classification of tweets containing self-reported covid-19 symptoms to address the tweet classification task we opted for fine-tuning two variants (cased and uncaseddeacc, which lowercases and removes accents) of robertuito -a roberta architecture language model (liu et al., 2019) pre-trained on social media text (pérez et al., 2021). the core element of the system presented for socialdisner task is the roberta-base-bne: roberta pre-trained on spanish general-domain corpus @xcite . ## experimental setup all models were fine-tuned on a single nvidia tesla v100 gpu by adding a linear classifier layer preceded by a 0.1 dropout layer on top of the original architecture using the hugging face transformers python library @xcite . in addition, to take the maximum advantage of the pre-trained model, we performed hyperparameter optimization that relied on the optuna framework @xcite . in the two-stage learning process, we performed optimization at each step within the same hyperparameter space which was also used for a singlestage fine-tuning. ## results our team submitted a total of two runs per task. in task 5 these corresponded to predictions made with cased model (run 1) and uncased-deacc model (run 2). only the best run was kept after the official evaluation. as for task 10, run 1 corresponds to the system based on the fine-tuned roberta model and run 2 to the one based on the same pre-trained model, but subjected to the-two stage fine-tuning procedure described in section 5. the metrics selected to evaluate performance of the participant systems are precision, recall and f1score for the self-report class or for each mention extracted where the spans overlap exactly in task 5 and task 10 respectively. the best performing system reached 0.77 f-1 score during in-house evaluation. while carrying out the evaluation and analysis of the results, we were able to determine that many miss-classified tweets contained an large number of spelling errors or incorporated direct speech citations. ## conclusions and future work this paper presents the participation of the sinai team in #smm4h workshop at coling 2022. firstly, we describe rule-based tweet preprocessing which was used in both tasks and aimed to normalize the peculiarities of the linguistic variety used in twitter,. for tweet classification task we compared transfer learning potential of two versions of roberta model pre-trained on social media text: cased and uncased-deacc. the latter performed better achieving 0.84 f1-score. to address the automatic disease mention recognition in tweets we employed a large pre-trained model and compared a state-of-the art fine-tuning approach with a weak-supervision enhanced twostage fine-tuning. for the tweet classification task, we compared transfer learning performance of two versions of language models pre-trained on social media text. the official evaluation for task 10 reveals that the model subjected to weak-supervision finetuning performs slightly better in terms of precision (0.739 vs 0.756) and f1-score (0.769 vs 0.775), which, however, cannot be considered a significant performance boost.