  Bidirectional Encoder Representations from Transformers (BERT) have shown to
be a promising way to dramatically improve the performance across various
Natural Language Processing tasks [Devlin et al., 2019]. Meanwhile, progress
made over the past few years by various Neural Net-work has also proved the
effectiveness of Neural Network in the field of Natural Language Processing. In
this project, RoBERTa-wwm-ext [Cui et al., 2019] pre-train language model was
adopted and fine-tuned for Chinese text classification. The models were able to
classify Chinese texts into two categories, containing descriptions of legal
behavior and descriptions of illegal behavior. Four different models are also
proposed in the paper. Those models will use RoBERTa-wwm-extas their embedding
layer and feed the embedding into different neural networks. The motivation
be-hind proposing these models is straightforward. By introducing complex
output layer architecture, the overall performance of the models could be
improved. All the models were trained on a data set derived from Chinese public
court records, and the performance of different models were compared.The
experiment shows that the performance of pro-posed models failed to beat the
original RoBERTa-wwm-ext model in terms of accuracy and training efficiency.
