S un B ear at WNUT -2020 Task 2: Improving BERT -Based Noisy Text Classification with Knowledge of the Data domain (2020)

Abstract: This paper proposes an improved custom model for WNUT task 2: Identification of Informative COVID-19 English Tweet. We improve experiment with the effectiveness of fine-tuning methodologies for state-of-the-art language model RoBERTa. We make a preliminary instantiation of this formal model for the text classification approaches. With appropriate training techniques, our model is able to achieve 0.9218 F1-score on public validation set and the ensemble version settles at top 9 F1-score (0.9005) and top 2 Recall (0.9301) on private test set.

## introduction since the outbreak of covid-19 pandemic, frequently updated information becomes a huge problem of concern. social media platforms consequently become real-time sources for news about flare-up data. in any case, the flare-up has been spreading quickly, we observe a monstrous amount of information on social networks, for example around 4 million covid-19 english tweets every day on twitter, in which most of these tweets are uninformative. therefore, it is crucial to collect the informative ones (for example corona virus tweets identified with new cases or dubious cases) for downstream applications. in any case, manual ways to deal with recognizing useful tweets require critical human endeavors, and hence are expensive. based on the dataset provided in wnut-2020 task 2: identification of informative covid-19 english tweets @xcite , we propose a fine-tuning strategy to adopt the universal language model roberta as an backbone model for text classification purposes. we also conduct several experiments in varied fine-tuning architectures on the pre-trained roberta. our best model results in a high f1-score of 0.9005 on the task's private test dataset and that of 0.9218 on the public validation set with multilayer perceptron head. ## related work one of the most important parts in text classification problems is input representation. traditional methods construct context-independent embeddings for words. mikolov et al. @xcite introduce an open-source word2vec, which consists of two models: continuous bag of words (cbow) and skip-gram model. the models were trained on 1.6 billion words to learn linguistic contexts of words. while word2vec is a selfsupervised algorithm, glove @xcite is trained unsupervised to form word embeddings. glove factorizes co-occurrence matrix of words, resulting in dense word vectors. however, both glove and word2vec fail representing rare or out-of-vocabulary words. fasttext @xcite mitigates this problem by decomposing words as a sum of character n-grams. this handles unseen words very well because these character n-grams may still occur in other words. in contrast to context-independent embeddings, modern language models encode word semantics within contexts. word vectors obtained from these methods achieve better results on downstream tasks because a word in different contexts expresses different meanings. bidirectional encoder representations from transformers @xcite , or bert for short, outperforms the previous best result with glue score of 80.4%, which is 7.6% improvement. there are two variants of bert: base and large; the large model is a stack of 24 transformers' encoders for a total of 340m parameters while the base one has only 12 encoders. gpt-2 @xcite by openai is a gigantic model with 1.5 billion parameters and 48 layers, setting new state-of-the-art results on 7 out of 8 datasets. facebook research team improves training procedures for bert, introducing roberta @xcite . the improvements include extended training time on a ten-times bigger dataset, increased batch size, using byte-level encoding with larger vocabulary, excluding next sentence predicting task, and dynamic masking pattern modifying. ## proposed method figure 1 illustrates our process. for mlm tuning we propose hierarchical tuning process that consists of two steps: domain adaptation using extra covid data and task adaptation using the given training data. after mlm tuning, we utilize different training techniques for text classification such as back translation, warm-up learning rate, layer freezing and layer-wise learning rates. this section provides details of this pipeline. ## conclusion in this paper, we have explored and proposed our pipeline to solve the identification of informative covid-19 english tweet task by using a pretrained universal language model. by conducting a lot of experiments, we have demonstrated that the use of roberta and our fine-tuning strategy is highly effective in text classification tasks. with our proposed methods, we have achieved prominent results on the wnut task 2. for future work, we will design more complex classification head architectures to improve model's performance as well as solving problems indicated in section 4.4. furthermore, we would like to employ our model and pipeline in different languages such as vietnamese to see how they adapt to new languages.