VTCC - NER at S em E val-2023 Task 6: An Ensemble Pre-trained Language Models for Named Entity Recognition (2023)

Abstract: We propose an ensemble method that combines several pre-trained language models to enhance entity recognition in legal text. Our approach achieved a 90.9873% F1 score on the private test set, ranking 2nd on the leaderboard for SemEval 2023 Task 6, Subtask B - Legal Named Entities Extraction.

## introduction named entity recognition (ner) is a key problem in natural language processing (nlp) that involves recognizing named entities in text and classifying them into specific types. recent studies on ner have achieved many promising results. most of state-of-the-art ner models are based on pretrained language models, which have been trained on large text corpora and are effective for word representation. however, domain-specific ner, such as legal ner, remains a challenging task. named entities in legal texts are more different and specific than named entities in general texts. for example, a person's name in legal texts may refer to a petitioner, respondent or lawyer. therefore, some pre-trained language models for legal domain have been released for use not only in legal ner, but also for other nlp tasks in legal texts. in this work, we propose a model that combines a pre-trained language model and some techniques, such as crf or dependency parsing, for ner in legal texts. an ensemble method is used to further boost the performance. dozier et al. (2010) defined * * corresponding author five legal named entities in us legal texts (judges, attorneys, companies, jurisdictions and courts) and developed a named entity recognition and resolution system based on pattern-based and statistical models. cardellino et al. (2017) trained a named entity recognizer, classifier and linker by mapping the lkif ontology to the yago ontology and developed a structured knowledge representation of legal concepts. due to variant writing style and vocabulary across different languages and forms, some datasets have been published to create separate models, such as german ler (leitner et al., 2020), legalnero (pȃis , et al., 2021), lener-br (luz de araujo et al., 2018), and legal ner @xcite . applying pre-trained language models boost the accuracy of ner tasks. bert @xcite is a language model based on transformer architecture, and was trained on two different tasks: masked language modeling (mlm) and next sentence prediction (nsp). roberta @xcite is a robust version of bert, and was trained on a much larger dataset and using a more effective training procedure. xlm-roberta @xcite is based on roberta architecture and was trained on the large multi-lingual dataset. deberta-v3 @xcite replaced mlm with replaced token detection (rtd) and implemented a gradient-disentangled embedding sharing method. deberta-v3 achieves state-of-the-art performance on many downstream nlp tasks. regarding the nlp tasks in legal texts including ner, some pre-trained language models have been trained exclusively in legal texts in order to enhance the performance. for instance, legal-bert @xcite , based on bert architecture, was trained on 12gb of diverse english legal texts and achieved better results than bert with the same number of parameters on some legal nlp tasks. inlegalbert and incaselegalbert @xcite were initialized from legal-bert and were trained on 27gb of indian legal documents. crf (collobert et al., 2011) is popular techniques used for ner. many studies have stacked them on top of other methods and achieved impressive results, as demonstrated by lample et al. (2016), ma and hovy (2016), peters et al. (2018), yang and zhang (2018), akbik et al. (2019). ## methodology for the legal ner subtask in semeval 2023 task 6 @xcite , the architecture of the proposed method is illustrated in the figure 1 . accordingly, we first execute the task with various benchmark pre-trained language models (plms) in this research field such as bert, robert, inlegal-bert, and so on. the output of aforementioned plms are then put into an ensemble module for extracting the final results. the processes of the proposed method are sequentially described as follows: ## conclusion to summarize, we applied two approaches to solve the named entity recognition of semeval 2023 task 6, subtask b. specifically, we applied an ensemble method to improve performance in the first approach. in contrast, the model did not perform well on the private test in the second approach. for the future work of this study, we try to optimize the weight for each plm in the ensemble module in order to improve the performance.