  Natural language processing is a fast-growing field of artificial
intelligence. Since the Transformer was introduced by Google in 2017, a large
number of language models such as BERT, GPT, and ELMo have been inspired by
this architecture. These models were trained on huge datasets and achieved
state-of-the-art results on natural language understanding. However,
fine-tuning a pre-trained language model on much smaller datasets for
downstream tasks requires a carefully-designed pipeline to mitigate problems of
the datasets such as lack of training data and imbalanced data. In this paper,
we propose a pipeline to adapt the general-purpose RoBERTa language model to a
specific text classification task: Vietnamese Hate Speech Detection. We first
tune the PhoBERT on our dataset by re-training the model on the Masked Language
Model task; then, we employ its encoder for text classification. In order to
preserve pre-trained weights while learning new feature representations, we
further utilize different training techniques: layer freezing, block-wise
learning rate, and label smoothing. Our experiments proved that our proposed
pipeline boosts the performance significantly, achieving a new state-of-the-art
on Vietnamese Hate Speech Detection campaign with 0.7221 F1 score.
