IXA at S em E val-2023 Task 2: Baseline Xlm-Roberta-base Approach (2023)

Abstract: IXA proposes a Sequence labeling fine-tune approach, which consists of a lightweight few-shot baseline (10e), the system takes advantage of transfer learning from pre-trained Named Entity Recognition and cross-lingual knowledge from the LM checkpoint. This technique obtains a drastic reduction in the effective training costs that works as a perfect baseline, future improvements in the baseline approach could fit: 1) Domain adequation, 2) Data augmentation, and 3) Intermediate task learning.

## introduction the tag set of multiconer is a fine-grained taxonomy 3 with 36 tags divided into 6 main groups. further relevant information related to semeval shared tasks is provided in guidelines @xcite . in addition to multilingualism, there were two key challenges in this year's task: (1) a fine-grained entity taxonomy which mainly penalizes the precision of systems as the ability to guess the correct tag, and (2) simulated errors added to the test set to make the task more realistic and difficult which mainly penalizes the recall of systems as the ability to guess the correct chunk of text to tag. ## related work semeval-2023 task 2 follows the multilingual ner task started in 2022 @xcite . critical challenges in the 2022 datasets were dealing with complex entities with limited context @xcite . this is commonly understood as a low-resourced specific domain problem e.g "medical entity recognition", main issues come from the fact that only a few of the total examples are annotated and the provided contexts show specific lexical usually unknown for language models e.g "in particular he worked on the laser photocoagulation of threshold retinopathy of prematurity". the challenges of sequence classification for recognizing complex entities in low-context conditions was recently outlined by @xcite . other work has extended the content to multilingual and code-mixed settings @xcite . in our experience, the multilingual constraints force the necessity of data to improve the learning results, here we identify three main paths: 1) domain adequation, which consists in learning mlm task with the raw texts provided in the task, 2) data augmentation, which is generating synthetic data with a sustainable lexical difference according to real examples but maintaining the possibility to align the annotated iob chains, and 3) intermediate task learning, which feeds language model with more expertise via silver dataset learning (similar tasks in other formats), this is usually pursued in more generic fine tune tasks such as natural language inference. ## results we pursued 12 experiments one per language, each xlm-roberta-base @xcite model was trained with 32 batch sizes for 10 epochs. the training process took on average around 25 minutes using nvidia tesla v100 gpu. we took around 13h using 400w that is 5.20 kw which could be understood as 7,84 co2e (lbs) generated by the experimentation. we provide two result tables to describe the proposed system performance, development evaluation shown in