D eep P avlov 1.0: Your Gateway to Advanced NLP Models Backed by Transformers and Transfer Learning (2024)

Abstract: We present DeepPavlov 1.0, an open-source framework for using Natural Language Processing (NLP) models by leveraging transfer learning techniques. DeepPavlov 1.0 is created for modular and configuration-driven development of state-of-the-art NLP models and supports a wide range of NLP model applications. DeepPavlov 1.0 is designed for practitioners with limited knowledge of NLP/ML. DeepPavlov is based on PyTorch and supports HuggingFace transformers. DeepPavlov is publicly released under the Apache 2.0 license and provides access to an online demo.

## introduction natural language processing (nlp) plays a critical role in many ai applications today, facilitating tasks such as automating customer service and processing large volumes of text data. however, the complexity of building, fine-tuning, and deploying state-of-the-art nlp models remains a significant barrier, particularly for non-experts in machine learning. the goal of nlp frameworks is to simplify and streamline the process of solving nlp tasks in software applications. these frameworks offer pre-built components, models, and tools for tasks such as named entity recognition (ner), sentiment analysis, text classification, and more. while the lifespan of nlp frameworks can be limited for various reasons, this does not imply that there is no space for nlp frameworks; instead, it highlights a growing gap in user-friendly, longterm nlp solutions. the continued success of an nlp framework relies not only on offering robust pre-trained models but also on responding to user needs by curating and expanding relevant datasets. active support and communication with the com-munity provides invaluable feedback and guides development in the right direction. deeppavlov @xcite ) framework, originally introduced at acl 2018, was designed to lower the entry barrier for non-experts, making advanced nlp accessible to a wider audience. originally built on tensorflow, it has since evolved with the release of deeppavlov 1.0, which now relies on pytorch and transformers while maintaining full backward compatibility in both philosophy and configuration files. deeppavlov 1.0 allow to utilize any autoconfing supported models. however, unlike transformers, deeppavlov 1.0 endorses no-code methodology, enabling the training, inference, and deployment exclusively through configuration files. by leveraging transformers, deep-pavlov 1.0 supports transfer learning techniques such as sequential transfer learning (stl), multi-task learning (mtl), and cross-lingual transfer learning (ctl). this enables the development of multilingual models and models that can handle scarce training data effectively. throughout its development, we realised that an nlp framework is not merely about pre-trained nlp models; it is equally important to continuously interact with users and address their needs. we concluded that to create a popular nlp framework, we must focus on gathering user-demanded datasets, thus, in section §5, we discuss the ner dataset that was compiled from openly available ner datasets and additionally augmented with synthetic data generated by large language models (llms). this dataset included custom location-related entities such as region, city, street_name, building_number, and apartment. the selection of entities was guided by user feedback, which was gathered through surveys conducted on our forum. our contribution can be summarized as follows: ## related work numerous frameworks are available to facilitate the development of nlp models. before discussing related work and comparable frameworks, it is important to first clarify what is similar but not directly relevant. although pytorch and tensorflow allow users to build nlp models, they require expertise in ml/nlp for training and deployment, placing them in a different category. while keras, pytorch lightning and transformers require less ml design knowledge from users, they still lack productionready nlp models and tools like docker and api interfaces, making them distinct from our focus. one could argue that in the realm of llms there might be less need for traditional nlp frameworks. models like chatgpt (openai, 2022) and gpt-4 (openai, 2023) demonstrate remarkable performance across a broad spectrum of nlp tasks. however, they have significant downsides and limitations. for instance, chatgpt often struggles with arithmetic, spatial, temporal, physical, and logical reasoning @xcite . additionally, deploying llms requires costly specialised hardware that many organizations cannot readily acquire. privacy concerns also arise, as sharing sensitive data with llm providers may violate user agreements or federal laws. in addition, encoder-based models fine-tuned with adequate training data typically surpass llms in a majority of nlp tasks, including ner @xcite , sentiment analysis @xcite , and qa @xcite . these limitations restrict the use of llms in real-world production environments. hence, frameworks like langchain foot_2 and others will not be included in our comparison for this work. a comparison of deeppavlov 1.0 with other nlp frameworks can be found in furthermore, flair has become a go-to testbed for nlp research, with projects such as flert @xcite and tars @xcite relying on it for evaluation. stanza foot_5 @xcite features a native python interface to the popular java-based stanford corenlp @xcite software, thus expanding its capabilities to include tasks such as coreference resolution and relation extraction. similarly to traditional nlp frameworks, stanza offers models for tokenization, lemmatization, pos tagging, dependency parsing, and ner. furthermore, stanza has been applied to biomedical and clinical text analysis, particularly for syntactic processing and ner tasks in these domains @xcite . below, we list related frameworks that are no longer supported: jiant foot_6 @xcite to deeppavlov in a sense that it supports multiple transfer learning techniques, such as sequential training and multi-task training, and also integrates with datasets and transformers to manage models and data. additionally, jiant supports glue @xcite , superglue @xcite and xtreme @xcite ) benchmarks. unfortunately, since october 17, 2021, the jiant project is no longer actively maintained. allennlp foot_7 @xcite is an open-source nlp library, created by ai2. initially, it gained widespread adoption due to its inclusion of pretrained elmo @xcite representations. as a result, it emerged as a popular nlp framework for tasks such as text understanding, information extraction, sentiment analysis, and reading comprehension. however, it is worth noting that the allennlp repository was archived and ownership transferred on december 16, 2022. ## design and implementation deeppavlov 1.0 adheres to the core model organization schema inherited from its predecessor, deeppavlov @xcite . to make it easier for newcomers, nlp models in deeppavlov are defined in separate configuration files, which include the parameters needed for training, inference, and deployment: dataset_reader, dataset_iterator, chainer, train, and metadata. sections dataset_reader and dataset_iterator are responsible for accessing the data and splitting it into training, validation, and test sets. dataset_reader supports datasets from huggingface. the chainer is a core concept of deeppavlov: it builds a pipeline from heterogeneous components (rule-based/ml/dl) and makes it possible to train or infer the entire pipeline as a unified unit. in addition, chainer specifies component inputs (in, in_y) and outputs (out) as arrays of names. a pipeline element can be either a function or an object of a class that implements __call__ method. any configuration file can be used within another configuration file as an element of the chainer, and any field of the nested configuration file can be overwritten. the train section defines training hyperparameters, such as trainer class, evaluation metrics, batch size, early stopping criteria, and many others. the metadata section contains variables used in other sections of the configuration file, for example, transformer encoder autoconfig alias, paths to pretrained model and corresponding dataset. deeppavlov allows users to easily customize the model configuration file. they can modify the hyperparameters, change the data preprocessing, or switch the classification model in the chainer while keeping the input and output format intact. for example, training the model on your own dataset is straightforward: just set the data_path in the dataset_reader to desired dataset location, and, if needed, adjust the training hyperparameters in the train section. a generalized schema of a deeppavlov's configuration file is depicted in figure 1 . ing pytorch's dataparallel. deeppavlov 1.0 incorporates huggingface transformers, enabling the utilization of all automodel transformer-based models from the huggingface hub. the framework offers versatile interaction with models through a command-line interface (cli), representational state transfer (rest), application programming interface (api), or python. deep-pavlov 1.0 can be installed by running pip install deeppavlov. the list of supported cli commands is as follows: install to install model-specific requirements, run python -m deeppavlov @xmath0, @xmath1. interact to get predictions from a model interactively through cli, run python -m deeppavlov @xmath2, where -d downloads files from the metadata of the configuration file (optional), and -i installs model requirements (optional). train to start the model training process, run the following command: python -m deeppavlov @xmath3. the dataset will be downloaded regardless of whether there was the -d flag or not. to train on custom data, users need to modify the dataset_reader path in the model configuration file. the data format is specified on the corresponding model documentation page. to change the architecture of the backbone transformer, users should modify the corresponding variable in the variables section. riseapi to run rest-like api server with the selected model, which might be useful for produc-tion usage, run python -m deeppavlov @xmath4. appendix a contains a code snippet that demonstrates how to interact with deeppavlov models. ## reference models deeppavlov 1.0 offers a range of fine-tuned nlp models for tasks such as text classification, token classification, and question answering. although many of the frameworks listed in token classification is traditionally represented by ner and pos models. based on our surveys, ner is the most popular model within deeppavlov. most nlp frameworks include ner models finetuned on openly available datasets, such as conll-2003 11 @xcite and ontonotes 12 (pradhan et al., 2013). based on user surveys, we consistently expand the range of ner entities by integrating data sets or generating synthetic data. our primary deep-pavlov 1.0 ner model 13 includes 32 entity types. the entity taxonomy is illustrated in figure 3 . the final ner model was trained on a combination of the ontonotes 5.0 and massive @xcite datasets. furthermore, we expanded these datasets with synthetic texts containing finegrained location-related entity types requested by our users, specifically street_name, state, region, city, building_number and apartment. to generate location-related entities, we employed openchat-3.5-0106 (7b) @xcite . using multiple prompt variations, we created a diverse collection of texts and manually in-10 topics_distilbert_base_uncased 11 ner_conll2003_bert 12 ner_ontonotes_bert 13 ner_bert_base spected them for any inconsistencies in the automatic markup. next, we merged the open datasets with synthetic texts by training two encoder-based models on each part individually and labelling the remaining parts. following the merge, we obtained a unified dataset incorporating all labels from both initial datasets. the deeppavlov 1.0 ner demo features 32 entities, which exceeds the number of entities in any other freely available ner dataset, to our knowledge, see along with a standard sentiment classification model, we developed an aspect-based sentiment analysis (absa) @xcite model in response to our users' requests. absa is a refined form of sentiment analysis that identifies aspects and their corresponding opinions within a given text. it has gained significant popularity in marketing because it offers more nuanced and targeted insights. specifically, we perform aspect-sentiment pair extraction (aspe) based on the input sequence s, where we label each token s i with sentiment polar-ities if applicable. in the example, 'i really like the interior but am disappointed with the dynamics of this car', 'interior' would be labelled as positive, and 'dynamics' as negative. the aspe formulation resembles a ner task with four classes: not an aspect, negative, neutral, and positive. as the base for our training set we utilized the semeval-2014 task 4 dataset @xcite , encompassing multi-domain data pertinent to absa. our proposed model attained micro-f1 scores of 68.8% and 80.5% for the laptop domain and restaurant domain accordingly. furthermore, as illustrated by the comparative analysis in @xcite , our methodology consistently surpasses chatgpt's performance across all evaluated scenarios. to our knowledge, none of the listed nlp frameworks in ## applications dream dialog system @xcite leverages deeppavlov mtl model fine-tuned on five classification tasks: emotion, sentiment, toxicity, intent, and topic. furthermore, the deep-pavlov's 1.0 ner model was enhanced to specifically handle issues related to the truecasing of au-14 https://hf.co/docs/transformers/model_doc/ auto tomatic speech recognition (asr) output, thereby facilitating its integration into a virtual assistant pipeline @xcite . in addition, we have been using deeppavlov 1.0 for fast prototyping while participating in semeval competitions, particularly, in multilingual shared task on hallucinations and related observable overgeneration mistakes (shroom) @xcite and in machine-generated text detection @xcite . ## conclusion and future work we introduced deeppavlov 1.0, an open-source nlp framework designed to streamline nlp model development and deployment. built on pytorch and transformers, deeppavlov 1.0 offers a diverse range of pre-trained nlp models accessible through api, cli, and python bindings. deep-pavlov 1.0 effectively addresses the challenge of limited training data by using transfer learning. this framework allows developers to focus on high-level implementation, reducing the need for extensive technical intricacies. to further assist users, deeppavlov 1.0 boasts an interactive online demo 15 and a dedicated forum 16 for support and collaboration. during the development of deeppavlov 1.0, we placed a strong emphasis on user needs. in our ongoing efforts, our aim is to enhance the framework by further expanding the ner model with custom entities. furthermore, we plan to construct a multilingual aspe dataset, broadening the scope and usability of deeppavlov 1.0.