MUCS @ T ech DO fication using F ine T uned Vectors and n-grams (2020)

Abstract: The increase in domain specific text processing applications are demanding tools and techniques for domain specific Text Classification (TC) which may be helpful in many downstream applications like Machine Translation, Summarization, Question Answering etc. Further, many TC algorithms are applied on globally recognized languages like English giving less importance for local languages particularly Indian languages. To boost the research for technical domains and text processing activities in Indian languages, a shared task named ”TechDOfication2020” is organized by ICON’20. The objective of this shared task is to automatically identify the technical domain of a given text which provides information about coarse grained technical domains and fine grained subdomains in eight languages. To tackle this challenge we, team MUCS have proposed three models, namely, DL-FineTuned model applied for all subtasks, and VC-FineTuned and VC-ngrams models applied only for some subtasks. n-grams and word embedding with a step of fine-tuning are used as features and machine learning and deep learning algorithms are used as classifiers in the proposed models. The proposed models outperformed in most of subtasks and also obtained first rank in subTask1b (Bangla) and subTask1e (Malayalam) with f1 score of 0.8353 and 0.3851 respectively using DL-FineTuned model for both the subtasks.

## introduction tc is one of the important areas of research in natural language processing (nlp). most of the tc experiments being conducted are on resource rich languages such as english, spanish etc. giving less importance to resource poor languages particularly indian languages. further, with the increase in domain specific text processing applications, domain specific tc is gaining importance demanding specialized tools and techniques to handle domain specific text datasets @xcite for many downstream applications like machine translation, summarization, question answering etc. in this direction, a shared task named "techdofication2020: technical domain identification" is organized in association with 17 th international conference on natural language processing foot_0 (icon) 2020 to automatically identify the technical domain of a given text in eight languages, namely, english, bangla, gujarati, hindi, malayalam, marathi, tamil, and telugu. these text provides information about specific coarse grained technical domains and fine grained subdomains. details of the shared tasks are provided in the task website foot_1 . technical domain identification can be modeled as a domain specific tc task. in this paper, we describe our models, namely, vc-ngrams, vc-finetuned and dl-finetuned proposed by our team mucs for technical domain identification in eight languages using n-grams and fine-tuned vectors as features. several machine learning (ml) and deep learning (dl) models are providing effective and accurate results for tc by reducing false positives @xcite . many dl models are using bidirectional long short term memory (bilstm) which contains two lstms: one taking the commitment to a forward course, and the other in a retrogressive way. bilstms are at the core of a few neural models accomplishing cutting edge execution in a wide assortment of undertakings in nlp @xcite . bilstm model can use the pre-trained word embeddings provided by fasttext and glove. the rest of the paper is organized as follows. section 2 highlights the related work followed by the proposed methodology in section 3. experiments and results are described in section 4 and the paper finally concludes in section 5. ## related work researchers round the globe have developed several approaches for domain specific tc. some of related ones are described below: a strategy to develop sentence vectors (sent2vec) by averaging the word embeddings is proposed by @xcite to explore the effect of word2vec on the performance of sentiments analysis of citations. the authors trained the acl-embeddings (300 and 100 measurements) from acl collection and also examined polarity-specific word embeddings (ps-embeddings) for characterizing positive and negative references. the generated embedding is fed to svm classifier and using 10-fold cross validation they obtained a macro-f1 score of 0.85 and the weighted-f1 score of 0.86 and proved the efficiency of word2vec on classifying positive and negative citations. a domain-specific intent classification for sinhala language proposed by @xcite utilized a feed-forward neural network with back propagation. they trained their model on a banking domain-related data set with mel frequency cepstral coefficients extracted from a sinhala speech corpus of 10 hours and obtained 74% results on identification accuracy of speech queries. @xcite proposed a combination of two models blstm-2d pooling and blstm-2d cnn and tested it on six tc tasks, including sentiment analysis, question classification, subjectivity classification, and newsgroups classification to compare their models with the state-of-the-art models. the authors evaluated the performance of proposed models on different lengths of sentences and then conducted a sensitivity analysis of 2d filter and max-pooling size. blstm-2dcnn model achieved excellent performance on 4 out of 6 tasks and obtained 52.4% and 89.5% test accuracies on stanford sentiment treebank-1 and treebank-2 datasets respectively. @xcite ) explored a multiclass tc task to classify uzbek language text. they collected articles on ten classes taken from the uzbek "daryo" online news and used six diverse ml algorithms namely, multinomial naıve bayes (mnb), decision tree classifier (dtc), svm, random forest (rf) and lr. hyper parameters for the classifiers were obtained by grid search algorithms with 5-fold cross-validation. the results obtained illustrates that, in many cases the character n-grams gives better results than the word level n-grams. however, among all,svm with the radial basis function kernel (rbf svm) using tf-idf and character level four grams features obtained best results with an accuracy of 86.88 ## methodology inspired by @xcite to use word embeddings as features for classification, @xcite to use bilstms for classification and @xcite for using ml algorithms for classification, we proposed three models, namely, voting classifiers using n-grams (vc-ngrams), voting classifiers using finetuned word vectors (vc-finetuned) and deep learning using finetuned word vectors (dl-finetuned). each model is built in two steps namely, i) feature extraction and ii) model construction, as explained below: the pre-trained word embeddings-glove, with a vector size 300 is used for english and bangla subtasks and fasttext with a size of 300 for the subtasks of rest of the languages as features after fine tuning using the training data. fine tuning of vectors in a pre-trained model by training data of a specific task helps in generating vectors for words missing in pre-trained models, and it can lead to higher performance specifically in fine-grained tasks such as domain specific tc. fine-tuned vectors are utilized to build embedding matrix to train vc-finetuned and dl-finetuned models.