Multichannel LSTM - CNN for T elugu Text Classification (2020)

Abstract: With the instantaneous growth of text information, retrieving domain-oriented information from the text data has a broad range of applications in Information Retrieval and Natural language Processing. Thematic keywords give a compressed representation of the text. Usually, Domain Identification plays a significant role in Machine Translation, Text Summarization, Question Answering, Information Extraction, and Sentiment Analysis. In this paper, we proposed the Multichannel LSTM-CNN methodology for Technical Domain Identification for Telugu. This architecture was used and evaluated in the context of the ICON shared task “TechDOfication 2020” (task h), and our system got 69.9% of the F1 score on the test dataset and 90.01% on the validation set.

## introduction technical domain identification is the task of automatically identifying and categorizing a set of unlabeled text passages/documents to their corresponding domain categories from a predefined domain category set. the domain category set consists of 6 category labels: bio-chemistry, communication technology, computer science, management, physics, and other. these domains can be viewed as a set of text passages, and test text data can be treated as a query to the system. domain identification has many applications like machine translation, summarization, question answering, etc. this task would be the first step for most downstream applications (i.e., machine translation). it decides the domain for text data, and afterward, machine translation can choose its resources as per the identified domain. majority of the research work in the area of text classification and domain identification has been done in english. there has been well below contribution for regional languages, especially indian languages. telugu is one of india's old traditional languages, and it is categorized as one of the dravidian language family. according to the ethnologue foot_0 list, there are about 93 million native telugu speakers, and it ranks sixteenth most-spoken languages worldwide. we tried to identify the domain of telugu text data using various supervised machine learning and deep learning techniques in our work. our multichannel lstm-cnn method outperforms the other methods on the provided dataset. this approach incorporates the advantages of cnn and self-attention based bilstm into one model. the rest of the paper is structured as follows: section 2 explains some related works of domain identification, section 3 describes the dataset provided in the shared task, section 4 addresses the methodology applied in the task, section 5 presents the results and error analysis, and finally, section 6 concludes the paper as well as possible future works. ## related work several methods for domain identification and text categorization have been done on indian languages, and few of the works have been reported on the telugu language. in this section, we survey some of the methodologies and approaches used to address domain identification and text categorization. @xcite explains the automatic text categorization with special emphasis on telugu. in his research work, supervised classification using the naive bayes classifier has been applied to 800 telugu news articles for text categorization. @xcite work on representing and categorizing indian language text documents using text mining techniques k nearest neighbour, naive bayes, and decision tree classifier. categorization of telugu text documents using language-dependent and independent models proposed by @xcite . @xcite introduced a model for document classification and text categorization. in their paper described a term frequency ontology-based text categorization for telugu documents. combining lstm and cnn's robustness, @xcite proposed attention-based multichannel convolutional neural network for text classification. in their network, bilstm encodes the history and future information of words, and cnn capture relations between words. ## dataset description we used the dataset provided by the organizers of task-h of techdofication 2020 for training the models. the data for the task consists of 68865 text documents for training, 5920 for validation, 2611 for testing. for hyperparameter tuning, we used the validation set provided by the organizers. the statistics of the dataset are shown in the text passages have been originally provided in the telugu script with the corresponding domain tags. the text documents have some noise, so before passing the text to the training stage, they are preprocessed using the following procedure: ## results and error analysis we first started our experiments with machine learning algorithms with various kinds of tf-idf feature vector representations. svm, mlp gave pretty good results on the validation dataset but failed on bio-chemistry and management data points. and cnn model with fasstext word embeddings seemed to be confused between the computer technology and cse data points (see the self-attention based bilstm model outperforms the cnn model on physics, cse, and computer technology data points though it performs worse on the bio-chemistry and management data points. if we observe the training set, 75% of the data samples belong to physics, cse, and computer technology domains remaining 25% of data owned by the remaining domain labels. so we were assuming that an imbalanced training set was also one of the problems for misclassification of biochemistry and management domain samples. we tried to handle the this data-skewing problem with smote @xcite technique, but it is doesn't work very well. after observing the cnn and bilstm model results, we ensemble these two models for better results. the ensemble multichannel lstm cnn model outperforms all our previous models, achieving a recall of 0.90 with the weighted f1-score of 0.90 on the development dataset and a recall of 0.698 with an f1-score of 0.699 on the test dataset (see ## conclusion in this paper, we proposed a multichannel approach that integrates the advantages of cnn and lstm. this model captures local, global dependencies, and sentiment in a sentence. our approach gives better results than individual cnn, lstm, and supervised machine learning algorithms on the telugu techdofication dataset. as discussed in the previous section, we will handle the data imbalance problem efficiently in future work. and we will improve the performance of the unambiguous cases in cse and computer technology domains.