Multi- BERT : Leveraging Adapters for Low-Resource Multi-Domain Adaptation (2025)

Abstract: Multi-domain text analysis presents significant challenges, particularly in Persian name entity recognition (NER). Using a single model for multiple domains often fails to capture the specific features of different domains. That is why many scientists have focused on prompting chatbots for this issue. However, studies show that these models do not achieve remarkable results in NER tasks without proper fine-tuning while training and storing a chatbot is extremely costly. This paper presents a new approach using one core model with various sets of domain-specific parameters. By using techniques like LoRAs and pre-fix tuning, along with extra layers, we train each set of trainable parameters for a specific domain. This allows the model to perform as well as individual models for each domain. Tests on various formal and informal datasets show that by using these added parameters, the proposed model performs much better than existing practical models. The model needs only one instance for storage but achieves excellent results across all domains. This paper also examines each adaptation strategy, outlining its strengths, weaknesses, and the best settings and hyperparameters for Persian NER. Lastly, this study introduces a new document-based domain detection system for situations where text domains are unknown. This novel pipeline enhances the adaptability and practicality of the proposed approach for real-world applications.

## introduction named entity recognition (ner) is an essential part of natural language processing (nlp) that helps in many tasks like information extraction or question answering. recently, ner has become even more important, thanks to the increased interest in nlp, which gave birth to many new challenges. one of the most pressing challenges is the adaptation of ner models to the ever-expanding text domains. this task becomes particularly difficult as model sizes and inference times increase. the dynamic nature of natural languages coupled with the diverse topics and contexts, presents a formidable challenge. especially, for languages like persian since there is a huge difference between texts found on formally written sites like wikipedia and informally written texts from social media posts, even when those posts are made from official pages like universities. when we look closer, it becomes evident that models trained within a specific textual domain often achieve worse results when confronted with data from other domains. this performance gap shows that many sentences require different entity labels based on their specific context. for instance, consider the sentence "tesla was robbed": in a scientific or historical context, "tesla" would likely be tagged as a person, whereas in discussions related to business or economics, or within the context of a casual tweet, "tesla" would be categorized as an organization. this ambiguity poses one of the primary challenges in accurately identifying entities, particularly within specialized fields such as medicine @xcite . traditionally, the preferred approach was to train a single transformer model for all domains. the model would be trained in a way that performs well in all domains. however, while we praise these models for performing well without knowing the text domain, these models tend to perform worse than models trained on a single domain. this problem has stimulated the exploration of different strategies to overcome the barriers posed by adapting to multiple domains in ner. an additional challenge associated with this approach arises from the limitation of a single model to produce outputs in only one format. certain domains may require different sets of labels, leading to the necessity for varied output formats tailored to each domain's needs. the adoption of multiple models to accommodate various domains introduces significant drawbacks, including resource-intensive requirements such as extensive ram and storage constraints, as well as the time-consuming process of training each instance. moreover, training a model for one task should inherently contribute to the understanding and performance improvement of another. to deal with these challenges many have turned to prompt engineering of generative models such as openai's chatgpt. however, studies show that without fine-tuning, these models underperform fine-tuned models by a large margin in many nlp tasks such as ner @xcite . furthermore, training these models requires a lot of computational resources. and a huge sum of data to train the model. this problem is extremely exacerbated when we look at the languages that are suffering from a lack of well-labeled and clean data such as the persian language. we delve deeper into the details of these models in section 2. therefore, a novel approach is proposed. this model offers an innovative solution to address these challenges by leveraging adapters. we incorporate specific parameters for each domain and create individual output layers to produce distinct outputs for each domain. subsequently, the added parameters and the output layers are trained for each domain. therefore, each set of these parameters and layers is only used for certain domains. this allows the model to perform perfectly in all domains. given access to a robust pre-trained model, the core model is frozen; however, it can also be trained during the training process if a pre-trained model is unavailable, the layers are shared between multiple domains, and the adapters are each only used for a single domain. remarkably, even when facing limited data availability, we observe a significant performance boost compared to other models. finally, we introduce an innovative approach for situations where the domain is unknown. the rest of this paper is organized as follows: section 2 gives a brief overview of the related projects that try to solve these issues. thereafter, section 3 explains the proposed architecture of the multi-bert model and the model will be thoroughly evaluated in section 4. moreover, section 5 will propose a novel pipeline that deals with the issues that arise from not knowing the exact domain of the texts, while section 6 concludes the paper and section 7 talks about the future possibilities. ## related work named entity recognition has always been a popular task in nlp. many new papers like punified-ner advocate for customizing and training a generative llm model capable of understanding diverse text domains and label sets @xcite . by incorporating a lot of information into prompts and leveraging extensive training data, this model demonstrates a remarkable capability to label various data types with diverse labels. however, other papers focus on the development of template-free models utilizing few-shot learning. these studies introduced models that, with a minimal set of labeled examples (typically 16, 32, or 64), can adeptly label texts @xcite @xcite . however, these papers still fine-tuned the model. in fact, it is shown that for some nlp tasks like ner fine-tuning the model is a crucial step and solely relying on prompt engineering results in subpar results @xcite . furthermore, our experiments with chatgpt on arman and parstwiner datasets have resulted in much worse performance compared to bert models. this is in line with other scientific research done with llms like abaskohi's benchmarking of chatgpt which achieved decent results on tasks like question answering while getting extremely low results on token classification tasks. @xcite . at the forefront of persian ner, current state-ofthe-art models include beheshtiner @xcite and parsbert @xcite . a bert model fine-tuned for ner tasks. recent advancements in persian ner have predominantly focused on fine-tuning models to cater to diverse domains. outstanding examples of this strategy include parstwiner @xcite , a bert model fine-tuned for informal and formal texts. however, as seen in this paper, while the model outperforms the original parsbert on informal text, it underperforms on the formal dataset, arman. another example is hengam, a bert model tailored for token classification in the tagging of formal and informal texts. as seen, these models exhibit notable drawbacks, such as prolonged training times and diminished performance in previous domains when adapting to new ones. the need for more efficient and adaptable models remains an ongoing concern within the landscape of persian ner research. ## the proposed method to mitigate the challenges posed by time and size constraints inherent in employing multiple models, a model called multi-bert is presented. in multi-bert, a single pre-trained model is completely frozen, while multiple sets of additional parameters and layers are integrated into the model. this configuration allows for the generation of diverse results from a single model. moreover, this design facilitates training for specific tasks without modifying the underlying base model, thereby safeguarding the performance of one task from affecting another. however, while the domain-specific parameters are completely separate from each other, we use pre-training for each set of parameters on other sets of data. this approach allows us to leverage any correlating information that can aid the model's performance. as seen in figure 1 , the model allows the selection of task-specific parameters during inference, tailoring the model's behavior to the requirements of each individual task. a significant advantage of multi-bert is its efficient use of adapters compared to traditional finetuning methods. adapters accelerate the training process @xcite , reducing the time required for each epoch and enabling model convergence in fewer than 10 epochs. this efficiency allows us to employ a two-step training approach: first, pre-training one set of parameters on all available data, and then fine-tuning a copy of these parameters for each specific task of interest. by leveraging task-specific data, we can effectively utilize cross-task information during fine-tuning. to incorporate these parameters we utilize adapters. after exploring various methods and adapters the most effective techniques were selected. the first chosen adapter is prefix-tuning @xcite . prefix tuning is a technique where a small set of learnable parameters, known as a prefix, is embedded directly into the input of all layers of a pre-trained language model. this allows the model to rapidly adapt to task-specific information without the need to fine-tune all the model parameters. the prefix acts as a continuous task-specific vector that can influence the behavior of the model across all layers, providing a lightweight and efficient way to customize large language models for specific tasks. it has been shown to achieve comparable performance to full model fine-tuning while requiring the tuning of only a tiny fraction of the parameters by leveraging the structured nature of prompts, this approach facilitates prompt-driven learning, a crucial aspect in multi-domain scenarios. on the other hand, for adapters, we employ the well-known low-rank adaptation method, also known as lora @xcite . this method yields comparable results by incorporating learnable parameters into the model layers. lora focuses on preserving adaptability without compromising the integrity of the base model. by adding parameters to each layer without introducing new ones, loras have emerged as highly reliable adapters. their efficiency lies in seamlessly integrating new parameters into existing layers, yielding impressive results within a short time frame, and facilitating straightforward merging of the new parameters with the existing layers. additionally, a classification layer is introduced based on the required number of classes. in cases where tasks share the same output structure, both the size of output and the specific labels, this layer can be shared among them. conversely, for tasks with differing output structures, we accommodate multiple final layers tailored to each task's unique requirements. this streamlined approach not only addresses the challenges associated with multiple models but also provides flexibility in adapting to diverse task requirements. the effectiveness of multi-bert is validated through comprehensive evaluations utilizing various parameter addition methods and task-specific classification layers. we use a fine-tuned core model, parsbert which is arguably the best pre-trained bert model. there are a lot of different models pre-trained for the persian language and each one can be used. however, based on our calculations parsbert performs the best for the ner tasks. therefore, parsbert was used in this study and due to the high performance of this model, it was frozen throughout the training steps of the multi-bert model. there are a few steps to train the model, firstly, for each domain, exactly one adapter is introduced, and for each set of adapters, that have the same output template, a classifier header is included. initially, one adapter is trained on all available data associated with that classifier excluding the domain of interest for a couple of epochs to ensure the adapters have all the correlating knowledge from other domains and the classifier is properly tuned. subsequently, this adapter is replicated across all other adapters of the same classifier and fine-tuned on each domain separately, until convergence and before over fitting. the classifier is only trained when the adapter is being pre-trained since this step includes all of the data associated with that particular output scheme. furthermore, this layer is frozen during the finetuning of the adapters this helps make the overall training to be shorter and helps preserve the knowledge of the adapters from inference. after the training process is complete, there is only one core model with multiple headers and many domainspecific parameters. therefore, we have multiple models each tailored for a single domain that can perform separately while they share much of their architecture. the subsequent section outlines the implementation and workflow of the proposed model, showing its efficiency and adaptability in handling multi-domain ner tasks. ## evaluation and results the proposed model is evaluated by using three different distinct datasets, each chosen to represent diverse unique challenges in the realm of ner and to test the performance of our model in dealing when faced with different challenges. we also introduce important baselines to show the effectiveness of our model. finally, we compare the results of all the models and discuss the hyperparameters and advantages of prefix-tuning and loras. ## document-based classifier pipeline in this section, we address the challenges posed by the absence of domain knowledge and propose an innovative solution to overcome this obstacle. fortunately, determining the domain of a given text becomes relatively straightforward when the text is sufficiently long or when multiple samples are available. nevertheless, feeding multiple samples simultaneously to a model is impractical, as it may lead to unwanted interference among distinct entries. to tackle this issue, we introduce a new pipeline by fine-tuning a new set of parameters to our core model. to achieve this objective, we aggregate every set of elements (for example 8 elements) and assign them a label representing the domain of the data. subsequently, we shuffle the data from all domains and train a new adapter for the model with the additional parameters tailored for a text classification task. upon completion of the training process, we construct our pipeline as seen in the figure 5 . when employing the model for inference-whether it involves tagging a series of comments on a website, tweets within a twitter thread, or processing a lengthy book-we provide 512 tokens from the text to the model to find the domain and based on the identified domain, we apply the respective parameters from the core model to obtain the final results. it is important to note that since we utilize the core model already employed in our token classification tasks and entirely freeze the core model during the classifier training, this pipeline does not adversely impact the main token classification models. to train, we concatenate each 8 input rows as one input. however, we only concatenate up to a length of 512. therefore, if the sum size of 5 elements exceeds 512 we only concatenate 5 elements and cut the first 512 tokens of it. consequently, each group of inputs turns into one input with the label of the dataset they are picked from. then, we mix and shuffle all of the data and train and evaluate the model on all the data. for the formal and the informal datasets, we get an accuracy of 100%. this is reasonable considering the distinctions between these two datasets. however, when we look at the parsner dataset we get an accuracy of 97% which is decent since we have 8 different classes. moreover, even if this model fails to predict the right domain of the text, it does not guarantee a wrong final output as the chosen version may generate correct results. in fact, the decided domain is probably extremely close for this mix-up to happen. hence we can use this pipeline to use the collection of texts to decide on their context and then process them normally one by one. ## conclusions this paper proposed the multi-bert model. this model is designed to perform well for all domains with any set of outputs. this is thanks to the deliberate design of this model by adding parameters for each domain and output layers for different sets of outputs coupled with the faster training time. this design allowed the model to perform the task on all domains perfectly. moreover, this paper evaluated the proposed model on arman (a formal dataset), parstwiner (an informal dataset), and the parsner, a collection of ten datasets from different contexts with large amounts of noise. the results proved that this model performs as well as the state-of-the-are for each domain, if not better. in addition, we also proposed a pipeline that can decide the domain of the data when a small set of sentences are available. we observed that if we use the model for sets of 8, it could understand the formality of the inputs completely with a 100% accuracy and can classify the exact domain of the news with an astonishing accuracy of 97% for the parsner dataset.