Fine-grained domain classification using Transformers (2020)

Abstract: The introduction of transformers in 2017 and successively BERT in 2018 brought about a revolution in the field of natural language processing. Such models are pretrained on vast amounts of data, and are easily extensible to be used for a wide variety of tasks through transfer learning. Continual work on transformer based architectures has led to a variety of new models with state of the art results. RoBERTa(CITATION) is one such model, which brings about a series of changes to the BERT architecture and is capable of producing better quality embeddings at an expense of functionality. In this paper, we attempt to solve the well known text classification task of fine-grained domain classification using BERT and RoBERTa and perform a comparative analysis of the same. We also attempt to evaluate the impact of data preprocessing specially in the context of fine-grained domain classification. The results obtained outperformed all the other models at the ICON TechDOfication 2020 (subtask-2a) Fine-grained domain classification task and ranked first. This proves the effectiveness of our approach.

## introduction the transformer-based language models have been showing promising progress on a number of different natural language processing (nlp) benchmarks. the combination of transfer learning methods with large-scale transformer language models is becoming a standard in modern nlp and has resulted in many state-of-the-art models. compared to lstms @xcite , the main limitations of bidirectional lstms is their sequential nature, which makes training in parallel very difficult. the transformer architecture solves that by completely replacing lstms by the so-called attention mechanism @xcite . with attention, we are seeing an entire sequence as a whole, therefore it is much easier to train in parallel. text classification is a well-known task in natural language processing, which aims at automatically providing additional document-level metadata (e.g. domain, genre, author). one of text classification tasks: domain classification can be divided into two categories:- ## dataset for the study, dataset from the icon techdofication 2020 (subtask-2a) was used. the entire collection consisted of 14910 text samples from english spanning across 7 sub-domains of computer science. a collection of 1929 text samples served as the blind test set for this task. ## bert vs roberta bert is a bi-directional transformer for pretraining over huge amount of unlabeled textual data to learn a language representation. it can be then used to fine-tune for specific machine learning tasks like text classification. bert outperformed the nlp state-of-the-art on several challenging tasks, attributed to the bidirectional transformer, novel pre-training tasks of masked language model @xcite and next sentence prediction @xcite . roberta has a very similar architecture as compared to bert with improved training methodology and more data. to improve the training, roberta removes the next sentence prediction task from bert's pre-training and introduces dynamic masking so that the masked token changes during the training epochs. originally bert is trained for 1m steps with a batch size of 256 sequences. roberta on the other hand is trained with 125 steps of 2k sequences and 31k steps with 8k sequences of batch size. large batches are also easier to parallelize via distributed parallel training. 4 system overview the section presents an overview of the system which was used to evaluate the scores described in the results section of the paper. ## results and evaluation all the models were evaluated on the dev dataset and the results are presented in in figure 1 (results on dev set without preprocessing), we can see that roberta miss classifies only slightly a less number text samples compared to bert with both performing very similar. however, there is difference seen when preprocessing is done and frequent words are removed. in figure 2 (results on dev set with preprocessing), we can see that bert miss-classifies 91 text samples as 'database management system' which reduces to 48 when using roberta. similarly, 87 miss-classifications done by bert as 'programming' are corrected to 54 by roberta. however it is seen that roberta tends to miss-classify text samples as 'software engineering' often. in both the cases, roberta performed better than bert however, with a small margin. the best performing model (roberta with preprocessing) was then evaluated using the icon techdofication 2020 (subtask-2a) test dataset. the results obtained are shown in ## conclusion in this paper, we did a comparative analysis of bert and roberta in the context of fine-grained domain classification. furthermore, the impact of pre-processing was also explored. it was found that pre-processing and removal of common terms from data helps the model perform better as more distinction is created between the sub-domains. the results indicate that roberta performs slightly better than bert in all the cases. the model proposed in this paper ranked first in the icon techdofication 2020 (subtask-2a) with an f1 score of 0.824.