E asy NLP : A Comprehensive and Easy-to-use Toolkit for Natural Language Processing (2022)

Abstract: Pre-Trained Models (PTMs) have reshaped the development of Natural Language Processing (NLP) and achieved significant improvement in various benchmarks. Yet, it is not easy for industrial practitioners to obtain high-performing PTM-based models without a large amount of labeled training data and deploy them online with fast inference speed. To bridge this gap, EasyNLP is designed to make it easy to build NLP applications, which supports a comprehensive suite of NLP algorithms. It further features knowledge-enhanced pre-training, knowledge distillation and few-shot learning functionalities, and provides a unified framework of model training, inference and deployment for real-world applications. EasyNLP has powered over ten business units within Alibaba Group and is seamlessly integrated to the Platform of AI (PAI) products on Alibaba Cloud. The source code of EasyNLP is released at GitHub ( https://github.com/alibaba/EasyNLP ).

## introduction pre-trained models (ptms) such as bert, gpt-3 and palm have achieved remarkable results in nlp. with the scale expansion of ptms, the performance of nlp tasks has been continuously improved; thus, there is a growing trend of ultra-largescale pre-training, pushing the scale of ptms from millions, billions, to even trillions @xcite @xcite . however, the application of large ptms in industrial scenarios is still a non-trivial problem, with reasons as follows. i) large ptms are not always smarter and can make commonsense mistakes due to the lack of world knowledge @xcite . hence, it is highly necessary to make ptms explicitly understand world facts by knowledgeenhanced pre-training, especially for supporting domain-specific applications. ii) although largescale ptms have achieved good results with few training samples, the problem of insufficient data and the huge size of models such as gpt-3 still restrict the usage of these models. thus, few-shot fine-tuning bert-style ptms is more practical for online applications @xcite . iii) last but not least, although large-scale ptms have become an important part of the nlp learning pipeline, the slow training and inference speed seriously affects online applications that require higher qps (query per second) with limited computational resources. to address these issues, we develop easynlp, an nlp toolkit that is designed to make the applications of large ptms to industrial scenarios more efficiently and effectively. easynlp provides knowledge-enhanced pre-training functionalities to improve the knowledge understanding abilities of ptms. specifically, it integrates our dkplm framework @xcite that enables the decomposition of knowledge-enhanced pre-training and task-specific learning. hence, the resulting models can be tuned and deployed in the same way as bert @xcite . easynlp is equipped with a variety of popular prompt-based few-shot learning algorithms such as pet @xcite and p-tuning @xcite . particularly, we propose a new fewshot learning paradigm named contrastive prompt tuning (cp-tuning) @xcite that eases the manual labor of verbalizer construction based on contrastive learning. finally, easynlp supports several knowledge distillation algorithms that compress large ptms into small and efficient ones. among them, the metakd algorithm @xcite can significantly improve the effectiveness of the learned models with cross-domain datasets, which is particular common in industry. overall, our easynlp toolkit can provide users with large-scale and robust learning functionalities, and is seamlessly connected to the platform of ai (pai) 1 products. to demonstrate the useful of easynlp, we also present the results of standard benchmarks and some real-world industrial scenarios to show how easynlp brings substantial improvements to these applications. in a nutshell, the main features of the easynlp toolkit include the following aspects: easynlp has rich apis to support the training of models from other open-source libraries such as huggingface/transformers 2 with the pai's distributed learning framework. it is also compatible with the ptms in easytransfer modelzoo 3 @xcite . easynlp also is equipped with knowledgeenhanced ptms of various domains. its pre-training apis enable users to obtain customized ptms using their own knowledge bases with just a few lines of codes. in this section, we summarize the related work on ptms, prompt learning and knowledge distillation. to name a few, bert @xcite learns bidirectional contextual representations by transformer encoders. other transformer encoder-based ptms include transformer-xl @xcite , xlnet @xcite and many others. the encoder-decoder and auto-regressive decoder architectures are used in t5 @xcite and @xcite . knowledge-enhanced ptms @xcite @xcite improve language understanding abilities of ptms via injecting relational triples extracted from knowledge bases. prompt learning for ptms. prompt learning models the probability of texts directly as the model prediction results based on language models @xcite . in the literature, pet @xcite ) models nlp tasks as cloze problems and maps the results of the masked language tokens to class labels. @xcite generates discrete prompts from t5 @xcite to support prompt discovery. p-tuning @xcite learns continuous prompt embeddings with differentiable parameters. our cp-tuning @xcite optimizes the output results based on contrastive learning, without defining mappings from outputs to class labels. knowledge distillation. knowledge distillation aims at learning a smaller model from an ensemble or a larger model @xcite . for large-scale ptms, distillbert @xcite and pkd @xcite applies the distillation loss in the pre-training and fine-tuning stages, separately. tinybert @xcite further distills bert in both stages, considering various types of signals. due to space limitation, we do not further elaborate other approaches. our metakd method @xcite further improves the accuracy of the student models by exploiting crossdomain transferable knowledge, which is fully supported by easynlp. ## the easynlp toolkit in this section, we introduce various aspects of our easynlp toolkit in detail. ## system evaluations and applications in this section, we empirically examine the effectiveness and efficiency of the easynlp toolkit on both public datasets and industrial applications. ## conclusion in this paper, we introduced easynlp, a toolkit that is designed to make it easy to develop and deploy deep nlp applications based on ptms. it supports a comprehensive suite of nlp algorithms and features knowledge-enhanced pre-training, knowledge distillation and few-shot learning functionalities for large-scale ptms. currently, easynlp has powered a number of business units inside alibaba cloud and provided nlp service on the cloud. the toolkit has been open-sourced to promote research and development for nlp applications.