fast H an: A BERT -based Multi-Task Toolkit for C hinese NLP (2021)

Abstract: We present fastHan, an open-source toolkit for four basic tasks in Chinese natural language processing: Chinese word segmentation (CWS), Part-of-Speech (POS) tagging, named entity recognition (NER), and dependency parsing. The backbone of fastHan is a multi-task model based on a pruned BERT, which uses the first 8 layers in BERT. We also provide a 4-layer base model compressed from the 8-layer model. The joint-model is trained and evaluated on 13 corpora of four tasks, yielding near state-of-the-art (SOTA) performance in dependency parsing and NER, achieving SOTA performance in CWS and POS. Besides, fastHan’s transferability is also strong, performing much better than popular segmentation tools on a non-training corpus. To better meet the need of practical application, we allow users to use their own labeled data to further fine-tune fastHan. In addition to its small size and excellent performance, fastHan is user-friendly. Implemented as a python package, fastHan isolates users from the internal technical details and is convenient to use. The project is released on Github.

## introduction recently, the need for chinese natural language processing (nlp) has a dramatic increase for many downstream applications. there are four basic tasks for chinese nlp: chinese word segmentation (cws), part-of-speech (pos) tagging, named entity recognition (ner), and dependency parsing. cws is a character-level task while others are word-level tasks. these basic tasks are usually the cornerstones or provide useful features for other downstream tasks. however, the chinese nlp community lacks an effective toolkit utilizing the correlation between the tasks. tools developed for a single task cannot achieve the highest accuracy, and loading tools for each task will take up more memory. in practical, there is a strong correlation between these four basic chinese nlp tasks. for example, the model will perform better in the other three word-level tasks if its word segmentation ability is stronger. recently, @xcite adopt cross-label to label the pos so that pos tagging and cws can be trained jointly. @xcite propose a graph-based model for joint cws and dependency parsing, in which a special "app" dependency arc is used to indicate the word segmentation information. thus, they can jointly train the word-level dependency parsing task and character-level cws task with the biaffine parser @xcite . @xcite explore adversarial multi-criteria learning for cws, proving more knowledge can be mined through training model on more corpora. as a result, there are many pieces of research on how to perform multi-corpus training on these tasks and how to conduct multi-task joint training. @xcite show the joint training of pos tagging and dependency parsing can improve each other's performance and so on. results of the cws task are contained in the output of the pos tagging task. therefore, we developed fasthan, an efficient toolkit with the help of multi-task learning and pretrained models (ptms) @xcite . fasthan adopts a bert-based @xcite jointmodel on 13 corpora to address the above four tasks. through multi-task learning, fasthan shares knowledge among the different corpora. this shared information can improve fasthan's performance on these tasks. besides, training on more corpora can obtain a larger vocabulary, which can reduce the number of times the model encounters characters outs of vocabulary. what's more, the joint-model can greatly reduce the occupied memory space. compared with training a model for each task, the joint-model can reduce the occupied memory space by four times. fasthan has two versions of the backbone model, base and large. the large model uses the first eight layers of bert, and the base model uses the theseus strategy @xcite to compress the large model to four layers. to improve the performance of the model, fasthan has done much optimization. for example, using the output of pos tagging to improve the performance of the dependency parsing task, using theseus strategy to improve the performance of the base version model, and so on. overall, fasthan has the following advantages: small size: the total parameter of the base model is 151mb, and for the large model the number is 262mb. ## backbone model the backbone of fasthan is a joint-model based on bert, which performs multi-task learning on 13 corpora of the four tasks. the architecture of the model is shown in figure 1 . for this model, sentences of different tasks are first added with corpus tags at the beginning of the sentence. and then the sentences are input into the bert-based encoder and the decoding layer. the decoding layer will use different decoders according to the current task: use conditional random field (crf) to decode in the ner task; use mlp and crf to decode in pos tagging and cws task; use the output of pos tagging task combined with biaffine parser to decode in dependency parsing task. each task uses independent label sets here, cws uses label set @xmath0, m, e, s}; pos tagging uses cross-labels set based on {b, m, e, s}; ner uses cross-labels set based on {b, m, e, s, o}; dependency parsing uses arc heads and arc labels to represent dependency grammar tree. ## fasthan fasthan is a chinese nlp toolkit based on the above model, developed based on fastnlp foot_0 and pytorch. we made a short video demonstrating fasthan and uploaded it to youtube foot_1 and bilibili foot_2 . fasthan has been released on pypi and users can install it by pip: pip install fasthan ## evaluation we evaluate fasthan in terms of accuracy, transferability, and execution speed. ## conclusion in this paper, we presented fasthan, a bert-based toolkit for cws, ner, pos, and dependency parsing in chinese nlp. after our optimization, fasthan has the characteristics of high accuracy, small size, strong transferability, and ease of use. in the future, we will continue to improve the fasthan with better performance, more features and more efficient learning methods, such as metalearning @xcite .