Word Embeddings, Cosine Similarity and Deep Learning for Identification of Professions & Occupations in Health-related Social Media (2021)

Abstract: ProfNER-ST focuses on the recognition of professions and occupations from Twitter using Spanish data. Our participation is based on a combination of word-level embeddings, including pre-trained Spanish BERT, as well as cosine similarity computed over a subset of entities that serve as input for an encoder-decoder architecture with attention mechanism. Finally, our best score achieved an F1-measure of 0.823 in the official test set.

## introduction during situations of risk, such as the covid-19 pandemic, detecting vulnerable occupations, be it due to their risk of direct exposure to the threat or due to mental health issues associated with work-related aspects, is critical to prepare preventive measures. these occupations can be detected through the analysis of tweets since twitter has become a very useful tool to find reliable information. due to the exponential growth of the use of this social network, natural language processing (nlp) techniques have become a crucial tool for unlocking this critical information. this paper describes the participation of our team in profner-st @xcite challenge, 7b subtrack of the sixth social media mining for health applications (smm4ha) @xcite , which focuses on the recognition of professions and occupations from twitter using spanish data. the core of the proposed system is based on an encoder-decoder architecture with attention mechanism successfully applied previously (ali and tan, 2019) for temporal expression recognition. this system combines several neural network architectures for the extraction of characteristics at a contextual level and a crf for the decoding of labels. the proposed system reach a f1 score of 0.823. ## results during experimentation our team apply the standard measures, precision, recall, and microaveraged f1-score, to evaluate the performance of our model. while the training set (miranda-escalada et al., 2020) was used for training the model, the development set was exploited to hyperparameter fine tuning. in the prediction stage, we combined both sets to training the model. the detailed hyper-parameter settings are illustrated in with the optimal parametric configuration obtained during the experimentation, the model obtains the results shown in the ## conclusion in these working notes we describe our proposed system based on an encoder-decoder architecture with an attention mechanism powered by a combination of word embeddings that include pre-trained fine-tuned spanish bert embeddings. future work would explore different data augmentation techniques as well as other entities information, as companies or organizations, which could contain important information related to occupations.