P ho BERT : Pre-trained language models for V ietnamese (2020)

Abstract: We present PhoBERT with two versions, PhoBERT-base and PhoBERT-large, the first public large-scale monolingual language models pre-trained for Vietnamese. Experimental results show that PhoBERT consistently outperforms the recent best pre-trained multilingual model XLM-R (Conneau et al., 2020) and improves the state-of-the-art in multiple Vietnamese-specific NLP tasks including Part-of-speech tagging, Dependency parsing, Named-entity recognition and Natural language inference. We release PhoBERT to facilitate future research and downstream applications for Vietnamese NLP. Our PhoBERT models are available at https://github.com/VinAIResearch/PhoBERT

## introduction pre-trained language models, especially bert @xcite -the bidirectional encoder representations from transformers @xcite , have recently become extremely popular and helped to produce significant improvement gains for various nlp tasks. the success of pre-trained bert and its variants has largely been limited to the english language. for other languages, one could retrain a language-specific model using the bert architecture @xcite @xcite or employ existing pre-trained multilingual bert-based models @xcite @xcite . in terms of vietnamese language modeling, to the best of our knowledge, there are two main concerns as follows: to handle the two concerns above, we train the first large-scale monolingual bert-based "base" and "large" models using a 20gb word-level vietnamese corpus. we evaluate our models on four downstream vietnamese nlp tasks: the common word-level ones of part-of-speech (pos) tagging, dependency parsing and named-entity recogni-tion (ner), and a language understanding task of natural language inference (nli) which can be formulated as either a syllable-or word-level task. experimental results show that our models obtain state-of-the-art (sota) results on all these tasks. our contributions are summarized as follows: ## phobert this section outlines the architecture and describes the pre-training data and optimization setup that we use for phobert. architecture: our phobert has two versions, phobert base and phobert large , using the same architectures of bert base and bert large , respectively. phobert pre-training approach is based on roberta @xcite which optimizes the bert pre-training procedure for more robust performance. pre-training data: to handle the first concern mentioned in section 1, we use a 20gb pre-training dataset of uncompressed texts. this dataset is a concatenation of two corpora: (i) the first one is the vietnamese wikipedia corpus (∼1gb), and (ii) the second corpus (∼19gb) is generated by removing similar articles and duplication from a 50gb vietnamese news corpus. 3 to task #training #valid #test pos tagging † 27,000 870 2,120 dep. parsing † 8,977 200 1,020 ner † 14,861 2,000 2,831 nli ‡ 392,702 2,490 5,010 ## experimental setup we evaluate the performance of phobert on four downstream vietnamese nlp tasks: pos tagging, dependency parsing, ner and nli. ## conclusion in this paper, we have presented the first largescale monolingual phobert language models pre-trained for vietnamese. we demonstrate the usefulness of phobert by showing that phobert performs better than the recent best multilingual model xlm-r and helps produce the sota performances for four downstream vietnamese nlp tasks of pos tagging, dependency parsing, ner and nli. by publicly releasing phobert models, we hope that they can foster future research and applications in vietnamese nlp.