Multilingual Pre-Trained Transformers and Convolutional NN Classification Models for Technical Domain Identification (2020)

Abstract: In this paper, we present a transfer learning system to perform technical domain identification on multilingual text data. We have submitted two runs, one uses the transformer model BERT, and the other uses XLM-ROBERTa with the CNN model for text classification. These models allowed us to identify the domain of the given sentences for the ICON 2020 shared Task, TechDOfication: Technical Domain Identification. Our system ranked the best for the subtasks 1d, 1g for the given TechDOfication dataset.

## introduction automated technical domain identification is a categorization/classification task where the given text is categorized into a set of predefined domains. it is employed in tasks like machine translation, information retrieval, question answering, summarization, and so on. in machine translation, summarization, question answering, and information retrieval, the domain classification model will help leverage the contents of technical documents, select the appropriate domain-dependent resources, and provide personalized processing of the given text. technical domain identification comes under text classification or categorization. text classification is one of the fundamental tasks in the field of nlp. text classification is the process of identifying the category where the given text belongs. automated text classification helps to organize unstructured data, which can help us gather insightful information to make future decisions on downstream tasks. traditional text classification approaches mainly focus on feature engineering techniques such as bag-of-words and classification algorithms @xcite . nowadays, the sate-of-the-art results on text classification are achieved by various nns such as cnn @xcite , lstm (hochreiter and schmidhuber, 1997), bert @xcite , and text gcn @xcite . attention mechanisms @xcite have been introduced in these models, which increased the representativeness of the text for better classification. transformer models such as bert @xcite uses the attention mechanism that learns contextual relations between words or sub-words in a text. text gcn @xcite uses a graphconvolutional network to learn a heterogeneous word document graph on the whole corpus, which helped classify the text. however, of all the deep learning approaches, transformer models provided sota results in text classification. in this paper, we present two approaches for technical domain identification. one approach uses the pre-trained multilingual bert model, and the other uses xlm-roberta with cnn model. the rest of the paper is structured as follows. section 2 describes our approach in detail. in section 3, we provide the analysis and evaluation of results for our system, and section 4 concludes our work. ## our approach here we present two approaches for the techdofication task. ## experiment this section presents the datasets used, the task description, and two models' performance on technical domain identification. we also include our implementation details and error analysis in the subsequent sections. ## error analysis the multilingual-bert model's confusion matrix is compared with the poorly performed model for languages, hindi, and tamil languages are shown in figure 3 . we chose hindi and tamil languages because, here, the difference in performance is more significant. for the hindi subtask, the svm classifier confused between "cse", "com tech", and "mgmt" labels, whereas the bert model performed better. for the tamil subtask, the svm classifier confused between "com tech" and "mgmt" labels, whereas the bert model performed better than the other models. this is because both the approaches (pre-trained multilingual-bert and pretrained xlm-roberta with cnn) learned better representation of the above data than the other models that helped in technical document identification. ## conclusion and future work we used pre-trained bi-directional encoder representations using multilingual-bert and xlm-roberta with cnn technical domain identification for english, bengali, gujarati, hindi, malayalam, marathi, tamil, and telugu languages. we compared the approaches with the baseline methods. our analysis showed that pre-trained multilingual bert and xlm-roberta with cnn models and finetuning it for text classification tasks showed an increase in macro f1 score and accuracy metrics compared to baseline approaches. some datasets are large, like for the hindi, tamil, and telugu, we can train the bert and xlm-roberta models from scratch and consider its hidden layer representation, and concatenate this with the representation of the pre-trained model. it might help to classify the datasets even better. sepp hochreiter and jürgen schmidhuber. 1997. long short-term memory. neural computation, 9(8):1735-1780. yoon kim. 2014. convolutional neural networks for sentence classification. arxiv preprint arxiv:1408.5882.