RIGA at S em E val-2023 Task 2: NER Enhanced with GPT -3 (2023)

Abstract: The following is a description of the RIGA team’s submissions for the English track of the SemEval-2023 Task 2: Multilingual Complex Named Entity Recognition (MultiCoNER) II. Our approach achieves 17% boost in results by utilizing pre-existing Large-scale Language Models (LLMs), such as GPT-3, to gather additional contexts. We then fine-tune a pre-trained neural network utilizing these contexts. The final step of our approach involves meticulous model and compute resource scaling, which results in improved performance. Our results placed us 12th out of 34 teams in terms of overall ranking and 7th in terms of the noisy subset ranking. The code for our method is available on GitHub ( https://github.com/emukans/multiconer2-riga ).

## introduction the semeval 2023 task 2: multiconer ii (multilingual complex named entity recognition) @xcite presented participants with a challenge to classify words into 36 named and nominal entities. unlike traditional named entity recognition (ner) tasks that deal with relatively simple classes such as persons, locations, and organizations, the multiconer task involves entities with any linguistic constituents. these include entries such as creative works, food, and medical procedures, among others. furthermore, rather than simply classifying a word as part of a high-level group, the task requires participants to choose the specific type within the group to which the word belongs. for instance, if the word pertains to a person, it should be classified as an artist, an athlete, a politician, and so on. the datasets provided for the multiconer ii task included 12 languages, with participants having the flexibility to submit their results for any language of their choosing. in our submission, we 1 https://github.com/emukans/multiconer2-riga focused solely on the most competitive english track and did not submit to other languages. the task at hand is considerably more complex when compared to traditional ner tasks, as all input texts are lower-cased. thus, our submission primarily relied on the utilization of pre-existing pre-trained large language models (llms). the history of machine learning compute can be divided into three eras, namely the pre-deep learning era, the deep learning era, and the large-scale era @xcite . the large-scale era was initiated by the release of alphago @xcite in late 2015. this trend is likely to have emerged due to large corporations identifying new opportunities in the field and making significant investments in it, thereby resulting in considerable increases in model training budgets. this, in turn, marked the beginning of a new era of large-scale language models. concurrently, it also prompted small institutes with limited budgets to explore novel methods for parameter efficient fine-tuning (peft) of llms developed by large corporations. several approaches have been proposed to address the issue of peft for llms. one such method involves incorporating small bottleneck layers referred to as adapters @xcite @xcite into the original llm. in this technique, all parameters of the original llm are frozen, and only the bottleneck layers are trainable. the approach involves adding just 3-5% of trainable parameters on top of the model, resulting in outcomes comparable to full fine-tuning. while it is true that full fine-tuning may yield marginally better results than adapters, we switched to full fine-tuning only at later steps. the task of selecting a specific entity type out of a group is a challenging one, even for humans, as it requires a considerable amount of background knowledge on various topics. without additional context about the sentence, correctly identifying the entity type by merely looking at a few words is nearly impossible. thus, in our submission, we focused on leveraging the gpt-3 model @xcite to enhance token class prediction. the authors of @xcite proposed a method called "in-context learning", which involves presenting the model with a task template with a blank gap that it must fill in. in our case, we utilized the gpt-3 language model to mine additional context about the sentences. as the gpt-3 language model is trained on data up to june 2021 (openai, 2023), its general knowledge is up to date, and it is aware of the most significant events up to that time. ## related work during semeval-2022 @xcite , the top-performing systems employed external knowledge bases @xcite and gazetteers @xcite to tackle the challenge. however, these solutions are prone to inaccuracies when faced with entities outside of the knowledge base and in scenarios where there are spelling errors and typos. in semeval-2023 @xcite , the authors highlight the limitations of the previous year's best-performing models in addressing these issues. recent advancements in natural language processing (nlp) have led to the development of transformer-based language models (lms) such as bert @xcite and, its larger and more advanced multi-lingual variant, xlm-r @xcite . these models have achieved further improvements in nlp tasks by utilizing deep contextual word representations. the difficulties of ner in real-world scenarios have been addressed in recent studies such as @xcite and @xcite . these studies employ external gazetteer knowledge bases to identify entities in low-context environments. typically, modern neural networks are trained using early stopping techniques @xcite , where the training process is halted when the loss function starts to increase on unseen validation data. in the past year, our team participated in the cod-woe task of semeval @xcite , where we explored various training effects such as "deep double descent" @xcite and model "scaling laws" @xcite . the "scaling law" effect occurs when two out of three training aspects, namely data, parameter count, and com-pute, are scaled. since the data amount was fixed, we scaled only the model size and compute time, and achieved an epoch-wise deep double descent. furthermore, we found that continuing training past the overfitting point can lead to more robust model predictions. in last year's competition, we submitted to several languages and achieved 1st place for french, 2nd place for spanish, and 3rd place for russian. these results demonstrate that loss is not always correlated with accuracy or f1 scores, and that stopping training too early can hurt performance. this year, we replicated our approach and scaled the number of trainable parameters and the amount of compute time. the results of training without early stopping have shown a significant improvement compared to the results obtained using early stopping. ## data in comparison to the previous iteration of the task, the new challenge in the latest dataset @xcite is the presence of noisy sentences. named entities in the dataset may contain typographical errors, missing named entity labels, or inaccurate labels. for instance, instead of being labeled as an athlete, the entity may be labeled as otherper, indicating that the entity is an unknown type of person. the multiconer2-data dataset was generated semi-automatically, which resulted in missing labels for some named entities. specifically, in some sentences that contained two named entities, the training and development datasets lacked labels for one or more of them. upon analysis of the multiconer2-data dataset, it was discovered that some of the sentences were duplicated. it is unclear whether the dataset creators intended to introduce bias into the model training or if the duplicates were accidental. the train dataset contains 0.7% duplicated data, while the test dataset contains 1.7% duplicated data. the complexity of the task is further compounded by the unbalanced frequency of named entities, as detailed in the distribution shown in the appendix a, and the size of the datasets, as indicated in as the size of the dev dataset is small, it is difficult to determine the significance of the difference in experimental results. therefore, another dataset, dataset size train validate dev test original 16.8k 0 0.9k 250k cleaned 13.3k 3.3k 0.9k 250k ## results we submitted two entries for the final ranking, one with additional contexts and the other without. the results of both entries are presented in regrettably, the organizers calculated the f1 scores for the clean and noisy subsets only for the top-performing solution. therefore, we have no information about these scores for the submissions without contexts. when comparing the submissions with and without contexts, a significant boost in performance is observed, with almost a 17-point increase. furthermore, it is concluded that gpt-3 is efficient in handling noise, leading to a small difference between the clean and noisy subsets. ranking based on the noisy subset elevates our score to the 7th position. we evaluated alternative language models, such as llama @xcite , which became available subsequent to the conclusion of the competition. however, the resultant contexts produced by llama exhibited comparatively lower levels of semantic coherence when contrasted with outputs derived from gpt-3. ## conclusion throughout the competition, our team confirmed the hypothesis that additional context significantly impacts the accuracy of named entity recognition. this highlights the importance for researchers to leverage existing large-scale language models, such as gpt-3, when designing solutions as it can greatly enhance overall performance and stability in a noisy environment. utilizing the scaling laws is one of the simplest ways to boost system performance. by scaling at least two out of three aspects of model training, the system outcome can be improved.