System description for P rof NER - SMMH : Optimized finetuning of a pretrained transformer and word vectors (2021)

Abstract: This shared task system description depicts two neural network architectures submitted to the ProfNER track, among them the winning system that scored highest in the two sub-tasks 7a and 7b. We present in detail the approach, preprocessing steps and the architectures used to achieve the submitted results, and also provide a GitHub repository to reproduce the scores. The winning system is based on a transformer-based pretrained language model and solves the two sub-tasks simultaneously.

## introduction the identification of professions and occupations in spanish (profner 1 , @xcite ) is part of the social media mining for health applications (smm4h) shared task 2021 @xcite . its aim was to extract professions from social media to enable characterizing health-related issues, in particular in the context of covid-19 epidemiology as well as mental health conditions. profner was the seventh track of the task and focused on the identification of professions and occupations in spanish tweets. it consisted of two sub-tasks: ## our approach we submitted two systems to each of the tasks described above, which share the same basic structure: in the backbone of both systems we take advantage of pretrained components, such as a transformerbased language model or skip-gram word vectors. the task head of both systems is very similar in that it solves task 7a and 7b simultaneously, and returns the sum of both losses. for the first system we aimed to maximize the metrics of the competition with the constraint of using a single gpu environment. for the second system we tried to maximize the model's efficiency with respect to the model size and speed while maintaining acceptable performance. both systems were designed and trained using biome.text foot_0 , a practical nlp open source library based on allennlp foot_1 @xcite and pytorch foot_2 @xcite . ## results with system 1 we managed to score highest on both profner 7a and 7b sub-tracks (f1:0.93/p:0.9251/r:0.933 and f1:0.839/p:0.838/r:0.84, respectively), with an average of 8 points above the arithmetic median of all submissions. the much smaller and faster (by a factor of ∼ 7) system 2 still manages to score above the competitions median (f1:0.88/p:0.9083/r:0.8553 and f1:0.764/p:0.815/r:0.718, respectively), but performs significantly worse when compared to system 1. we find a clear correlation between the classification f1 score and the f1 score of the ner task in our hpo runs, which signals that the feedback loop between the two tasks is in general beneficial and advocates solving both tasks simultaneously. when comparing system 1 and 2, it seems that the amount of training data provided to the rnn architecture was not sufficient to match the transfer capabilities of the pretrained transformer, even with dedicated architecture searches and extensive hyperparameter tuning. this is corroborated by the fact that adding the validation data to the training data led to a clear performance boost for system 2, while the performance of system 1 stayed almost the same (compare the f1 test and validation metrics for task 7b in a possible path to improve system 1, which was not pursued due to time constraints, could be the inclusion of the gazetteers provided during the profner track. we consider this path especially promising given the fact that the precision was always lower than the recall for both tasks. we conclude that the exploitation of the transfer capabilities of a pretrained language model and its optimized fine tuning to the target domain, provides an conceptually easy system architecture and seems to be the most straight forward method to achieve competitive performance, especially for tasks where training data is scarce. to help to reproduce our results, we provide a github repository at https://github.com/ recognai/profner .