P ho NLP : A joint multi-task learning model for V ietnamese part-of-speech tagging, named entity recognition and dependency parsing (2021)

Abstract: We present the first multi-task learning model – named PhoNLP – for joint Vietnamese part-of-speech (POS) tagging, named entity recognition (NER) and dependency parsing. Experiments on Vietnamese benchmark datasets show that PhoNLP produces state-of-the-art results, outperforming a single-task learning approach that fine-tunes the pre-trained Vietnamese language model PhoBERT (Nguyen and Nguyen, 2020) for each task independently. We publicly release PhoNLP as an open-source toolkit under the Apache License 2.0. Although we specify PhoNLP for Vietnamese, our PhoNLP training and evaluation command scripts in fact can directly work for other languages that have a pre-trained BERT-based language model and gold annotated corpora available for the three tasks of POS tagging, NER and dependency parsing. We hope that PhoNLP can serve as a strong baseline and useful toolkit for future NLP research and applications to not only Vietnamese but also the other languages. Our PhoNLP is available at https://github.com/VinAIResearch/PhoNLP

## introduction vietnamese nlp research has been significantly explored recently. it has been boosted by the success of the national project on vietnamese language and speech processing (vlsp) kc01.01/2006-2010 and vlsp workshops that have run shared tasks since 2013. 1 fundamental tasks of pos tagging, ner and dependency parsing thus play important roles, providing useful features for many downstream application tasks such as machine translation @xcite , sentiment analysis @xcite , relation extraction (to and do, 2020), semantic parsing @xcite , open information extraction @xcite and question answering @xcite . thus, there is a need to develop nlp toolkits for linguistic annotations w.r.t. vietnamese pos tagging, ner and dependency parsing. vncorenlp @xcite is the previous public toolkit employing traditional featurebased machine learning models to handle those vietnamese nlp tasks. however, vncorenlp is now no longer considered state-of-the-art because its performance results are significantly outperformed by ones obtained when fine-tuning phobert-the current state-of-the-art monolingual pre-trained language model for vietnamese @xcite . note that there are no publicly available fine-tuned bert-based models for the three vietnamese tasks. assuming that there would be, a potential drawback might be that an nlp package wrapping such fine-tuned bertbased models would take a large storage space, i.e. three times larger than the storage space used by a bert model @xcite , thus it would not be suitable for practical applications that require a smaller storage space. jointly multi-task learning is a promising solution as it might help reduce the storage space. in addition, pos tagging, ner and dependency parsing are related tasks: pos tags are essential input features used for dependency parsing and pos tags are also used as additional features for ner. jointly multi-task learning thus might also help improve the performance results against the single-task learning @xcite . in this paper, we present a new multi-task learning model-named phonlp-for joint pos tagging, ner and dependency parsing. in particular, given an input sentence of words to phonlp, an encoding layer generates contextualized word embeddings that represent the input words. these contextualized word embeddings are fed into a pos tagging layer that is in fact a linear prediction layer @xcite to predict pos tags for the corresponding input words. each predicted pos tag is then represented by two "soft" embeddings that are later fed into ner and dependency parsing layers separately. more specifically, based on both the contextualized word embeddings and the "soft" pos tag embeddings, the ner layer uses a linear-chain crf predictor @xcite to predict ner labels for the input words, while the dependency parsing layer uses a biaffine classifier @xcite to predict dependency arcs between the words and another biaffine classifier to label the predicted arcs. our contributions are summarized as follows: ## model description figure 1 illustrates our phonlp architecture that can be viewed as a mixture of a bert-based encoding layer and three decoding layers of pos tagging, ner and dependency parsing. ## phonlp toolkit we present in this section a basic usage of our phonlp toolkit. we make phonlp simple to setup, i.e. users can install phonlp from either source or pip (e.g. pip3 install phonlp). we also aim to make phonlp simple to run from both the command-line and the python api. for example, annotating a corpus with pos tagging, ner and dependency parsing can be performed by using a simple command as in figure 2 . assume that the input file "input.txt" in figure 2 contains a sentence "tôi đang làm_việc tại python3 run_phonlp.py --save_dir ./pretrained_phonlp --mode annotate --input_file input.txt --output_file output.txt 1 tôi p o 3 sub 2 đang r o 3 adv 3 làm_việc v o 0 root 4 tại e o 3 loc 5 vinai np b-org 4 pob 6 . vinai ." (i tôi am đang working làm_việc at tại vinai).