An Attention Ensemble Approach for Efficient Text Classification of I ndian Languages (2020)

Abstract: The recent surge of complex attention-based deep learning architectures has led to extraordinary results in various downstream NLP tasks in the English language. However, such research for resource-constrained and morphologically rich Indian vernacular languages has been relatively limited. This paper proffers a solution for the TechDOfication 2020 subtask-1f: which focuses on the coarse-grained technical domain identification of short text documents in Marathi, a Devanagari script-based Indian language. Availing the large dataset at hand, a hybrid CNN-BiLSTM attention ensemble model is proposed that competently combines the intermediate sentence representations generated by the convolutional neural network and the bidirectional long short-term memory, leading to efficient text classification. Experimental results show that the proposed model outperforms various baseline machine learning and deep learning models in the given task, giving the best validation accuracy of 89.57% and f1-score of 0.8875. Furthermore, the solution resulted in the best system submission for this subtask, giving a test accuracy of 64.26% and f1-score of 0.6157, transcending the performances of other teams as well as the baseline system given by the organizers of the shared task.

## introduction the advent of attention-based neural networks and the availability of large labelled datasets has resulted in great success and state-of-the-art performance for english text classification @xcite @xcite . such results, however, for indian language text classification tasks are far and few as most of the research employ traditional machine learning and deep learning models @xcite @xcite . apart from being heavily consumed in the print format, the growth in the indian languages internet user base is monumental, scaling from 234 million in 2016 to 536 million by 2021 foot_0 . even so, just like most other indian languages, the progress in nlp for marathi has been relatively constrained, due to factors such as the unavailability of large-scale training resources, structural un-similarity with the english language, and a profusion of morphological variations, thus, making the generalization of deep learning architectures to languages like marathi difficult. this work posits a solution for the techdofication 2020 subtask-1f: coarse-grained domain classification for short marathi texts. the task provides a large corpus of marathi text documents spanning across four domains: biochemistry, communication technology, computer science, and physics. efficient domain identification can potentially impact, and improve research in downstream nlp applications such as question answering, transliteration, machine translation, and text summarization, to name a few. inspired by the works of @xcite @xcite , a hybrid cnn-bilstm attention ensemble model is proposed in this work. in recent years, convolutional neural networks @xcite @xcite @xcite and recurrent neural networks @xcite have been used quite frequently for text classification tasks. quite different from one another, the cnns and the rnns show different capabilities to generate intermediate text representation. cnn models an input sentence by utilizing convolutional filters to identify the most influential n-grams of differ-ent semantic aspects @xcite . rnn can handle variable-length input sentences and is particularly well suited for modeling sequential data, learning important temporal features and longterm dependencies for robust text representation @xcite . however, whilst cnn can only capture local patterns and fails to incorporate the long-term dependencies and the sequential features, rnn cannot distinguish between keywords that contribute more context to the classification task from the normal stopwords. thus, the proposed model hypothesizes a potent way to subsume the advantages of both the cnn and the rnn using the attention mechanism. the model employs a parallel structure where both the cnn and the bilstm model the input sentences independently. the intermediate representations, thus generated, are combined using the attention mechanism. therefore, the generated vector has useful temporal features from the sequences generated by the rnn according to the context generated by the cnn. results attest that the proposed model outperforms various baseline machine learning and deep learning models in the given task, giving the best validation accuracy and f1-score. ## related work since the past decade, the research in nlp has shifted from a traditional statistical standpoint to complex neural network architectures. the cnn and rnn based architectures have emerged greatly successful for the text classification task. yoon kim was the first one who applied a cnn model for english text classification. in this work, a series of experiments were conducted with single as well as multi-channel convolutional neural networks, built on top of randomly generated, pretrained, and fine-tuned word vectors @xcite . this success of cnn for text classification led to the emergence of more complex cnn models @xcite as well as cnn models with character level inputs @xcite . rnns are capable of generating effective text representation by learning temporal features and long-term dependencies between the words @xcite . however, these methods treat each word in the sentences equally and thus cannot distinguish between the keywords that contribute more to the classification and the common words. hybrid models proposed by @xcite and @xcite succeed in exploiting the advantages of both cnn and rnn, by using them in combination for text classification. since the introduction of the attention mechanism @xcite , it has become an effective strategy for dynamically learning the contribution of different features to specific tasks. needless to say, the attention mechanism has expeditiously found its way into nlp literature, with many works effectively leveraging it to improve the text classification task. @xcite proposed a cnn -rnn attention-based neural network (cran) for text classification. this work illustrates the effectiveness of using the cnn layer as a context of the attention model. results show that using this mechanism enables the proposed model to pick the important words from the sequences generated by the rnn layer, thus helping it to outperform many baselines as well as hybrid attention-based models in the text classification task. @xcite proposed an attention pooling strategy, which focuses on making a model learn better sentence representations for improved text classification. authors use the intermediate sentence representations produced by a bilstm layer in reference with the local representations produced by a cnn layer to obtain the attention weights. experimental results demonstrate that the proposed model outperforms state-of-the-art approaches on a number of benchmark datasets for text classification. @xcite combine the bilstm and cnn with the attention mechanism for fine-grained text classification tasks. the authors employ a method in which intermediate sentence representations generated by bilstm are passed to a cnn layer which is then max pooled to capture the local features of a sentence. the local feature representations are further combined by using an attention layer to calculate the attention weights. in this way, the attention layer can assign different weights to features according to their importance to the text classification task. the literature in nlp focusing on the resourceconstrained indian languages has been fairly restrained. @xcite ing approaches for hindi sentence classification. the authors have evaluated the effect of using pretrained fasttext hindi embeddings on the sentence classification task. the finest classification performance is achieved by the vanilla cnn model when initialized with fasttext word embeddings fine-tuned on the specific dataset. ## dataset the techdofication-2020 subtask-1f dataset consists of labelled marathi text documents, each belonging to one of the four classes, namely: biochemistry (bioche), communication technology (com tech), computer science (cse), and physics (phy). the training data has a mean length of 26.89 words with a standard deviation of 25.12. ## proposed model this section describes the proposed multi-input attention-based parallel cnn-bilstm. figure 1 depicts the model architecture. each component is explained in detail as follows: ## experimental setup each text document is tokenized and padded to a maximum length of 100. longer documents are truncated. the work of @xcite is referred for selecting the optimal set of hyperparameters for training the fasttext skip-gram model. the 300-dimensional fasttext word embeddings are trained on the given corpus for 50 epochs, with a minimum token count of 1, and 10 negative examples, sampled for each instance. the rest of the hyperparameter values were chosen as default @xcite . after training, an average loss of 0.6338. was obtained over 50 epochs. the validation dataset is used to tune the hyperparameters. the lstm layer dimension is set to 128 neurons with a dropout rate of 0.3. thus, the bil-stm gives an intermediate representation of 256 dimensions. for the cnn block, we employ three parallel convolutional layers with filter sizes 3, 4, and 5, each having 256 feature maps. a dropout rate of 0.3 is applied to each layer. the local representations, thus, generated by the parallel cnns are then concatenated and max-pooled. all other parameters in the model are initialized randomly. the model is trained end-to-end for 15 epochs, with the adam optimizer (kingma and ba, 2014), sparse categorical cross-entropy loss, a learning rate of 0.001, and a minibatch size of 128. the best model is stored and the learning rate is reduced by a factor of 0.1 if validation loss does not decline after two successive epochs. ## baseline models the performance of the proposed model is compared with a host of machine learning and deep learning models and the results are reported in feature based models: multinomial naive bayes with bag-of-words input (mnb + bow), multinomial naive bayes with tf-idf input (mnb + tf-idf), linear svc with bag-of-words input (lsvc + bow), and linear svc with tf-idf input (lsvc + tf-idf). @xcite , and serial bilstm-cnn + attention. ## results and discussion the performance of all the models is listed in