UMUT eam@ T amil NLP - ACL 2022: Emotional Analysis in T amil (2022)

Abstract: This working notes summarises the participation of the UMUTeam on the TamilNLP (ACL 2022) shared task concerning emotion analysis in Tamil. We participated in the two multi-classification challenges proposed with a neural network that combines linguistic features with different feature sets based on contextual and non-contextual sentence embeddings. Our proposal achieved the 1st result for the second subtask, with an f1-score of 15.1% discerning among 30 different emotions. However, our results for the first subtask were not recorded in the official leader board. Accordingly, we report our results for this subtask with the validation split, reaching a macro f1-score of 32.360%.

## introduction in this work, we detail the participation of the umuteam in the shared-task tamil nlp (acl 2022), concerning emotion analysis (ea) in tamil @xcite . emotion detection is a recent field of research included in the broader research area of sentiment analysis. here, the target of emotion detection aims at detecting types of feelings in natural language like anger, fear, disgust, happiness, surprise and sadness @xcite . in literature, strategies can be found addressing emotion detection in quite different domains. for instance, @xcite propose an architecture based on leaky relu activated deep neural network (lra-dnn) to address emotion analysis on social media. the architecture is comprised of four steps: (1) preprocessing to clean the data and change it representation in a more understandable format; (2) feature extraction step to extract the most relevant characteristics; (3) ranking step where extracted features are assigned to ranks that they are optimised by using a nature-inspired meta-heuristic optimisation algorithm; and (4) classification where the lra-dnn is employed. yong et al., @xcite describe a bcblac model designed to tackle emotion analysis in a food review. its name is due to its layer architecture: bert layer, cnn layer, blstm layer, attention layer and crf layer. each layer represents a step that the input must go through to carry out the emotion classification process. in this shared task, the organisers challenged the participants to extract one emotion per document from a collection of social media comments written in tamil. the organisers provided the participants with three sets: development, training and test. it is worth mentioning that we use these splits as expected, that is, we train with the training set and validate with the development set. this shared task was divided into two minor subtasks. the first subtask distinguishes among 11 emotions whereas the second subtask with 30 emotions. the name of the emotions, and the number of instances per training and validation are depicted in figure 1 . our research group has experience dealing with ea tasks. for example, we participated in the emoevales shared task, proposed in iberlef 2021 @xcite concerning ea in spanish. this task consisted into a multi classification task with the ekman basic emotions. we achieved the 6th position with an accuracy of 68.5990% (4.1667% below the best result) @xcite . in this shared-task we participated with similar methods to the ones described in @xcite . however, here we conduct a more advance hyperparameter tuning stage. besides, we use this task to validate a subset of language-independent linguistic features extracted with a custom tool that is part of the doctoral thesis of one the members of the team. in fact, we had participated in different automatic document classification tasks in spanish to validate these linguistic features. we have observed that these linguistic features contribute to improve state-of-the-art models based on transformers. one of the secondary objectives of our participation is therefore to observe whether the subset of these linguistic features that are language-independent still improve the performance of automatic document classification in non-latin languages. this subset of linguistic features are based on stylometry, which different metrics concerning word and sentence length as well as punctuation symbols. there are, in addition, features that capture emojis, hyperlinks, and social network jargon. ## methodology in a nutshell, our participation consists in the development of a classifier based on neural network that uses four feature sets combined using a knowledge integration strategy. during the development stage, other methods for combining these features, such as ensemble learning, are evaluated. next, we describe the four feature sets in detail. the first feature set is lf, a subset of language-independent linguistic features extracted using the umutextstats tool @xcite . these features are stylometric features, pos features based on the tamil model of stanza @xcite , and social media features that includes the detection of emojis. the second feature set is se, that are non-contextual sentence embeddings from the tamil pretrained model from fasttext @xcite . the third and forth feature sets are bf and rf. these features are, respectively, sentence embeddings from multilingual bert @xcite and multilingual roberta @xcite . to obtain the sentence embeddings from bert and roberta, we fine-tuned them separately for each task with raytune @xcite . during this stage, 10 models with tree of parzen estimators (tpe) @xcite were trained to obtain the optimum values for the (1) weight decay, (2) batch size, (3) warm-up speed, (4) number of epochs, and (5) learning rate. tpe strategy selects the next hyperparameters using bayesian reasoning and the expected improvement. next, we extract the [cls] token from the best models in a similar way as described in @xcite . next, we train a neural network per feature set separately. we use these neural networks to build two classifiers based on ensemble learning. for this, we use keras (tensorflow) and raytune for the hyperparameter stage. besides, we train another neural network that combines all the feature sets at once using a knowledge integration strategy. for this, we fed each feature set in a separate hidden layer and then combine their outputs in the hidden layers. the details of the hyperparameter optimisation stages are the following. as all feature sets are of a fixed size, we evaluate only multilayer perceptrons (mlp) as the network architecture. these mlps are divided into shallow and deep neural networks. this category is based on the number of hidden layers and the number of neurons per layer. specifically, for the shallow neural networks we only try one or two hidden layers maximum. the number of neurons is the same in all layers. in deep neural networks, however, we try a larger number of hidden layers (between 3 and 8). besides, the number of neurons per layer are arranged in different shapes (brick, triangle, diamond, rhombus, and short and long funnel). we also try several activation functions to connect the hidden layers as well as several learning rates and ratios of a dropout mechanism. we also handle class imbalance evaluating larger batch sizes and class weights. the best configuration for the knowledge integration strategy for subtask 1 is a deep neural network composed of 3 hidden layers, with 128 neurons stacked in a triangle shape. the batch size is 64, the dropout of .2, the learning rate is 0.001, and the activation function that connects the layers is a sigmoid. on the other hand, the best configuration for subtask 2 is a batch size of 32, no dropout, 4 hidden layers with 57 neurons stacked with a rhombus shape (the value of 57 is the max value of neurons per hidden layer), a learning rate of 0.001, and selu as activation function. ## results and discussion as it can be observed from the first subtask (see in view of these results, and as we only have one chance to submit our proposal, we decided to send the results with the knowledge integration strategy because, in our experience, it tends to produce better results with unseen test splits. the classification report for the second subtask is depicted in the results are also lower than the best of the feature sets in isolation: bf (macro f1-score of 13.38). in this case, the macro f1-score of combining the features using ensembles are 12.98% for the mode, and 15.12% for averaging the predictions of each model. as it can be observed, the result with the ensemble learning outperforms both the results achieved with bf and the combination of features into the same neural network. however, we decided to send the results using the same strat- ## conclusions and further research lines in this working notes we have described the participation of the umuteam in a shared task regarding emotion analysis in tamil. we achieved the 1st position in a fine-grained emotion analysis classification in which 30 emotions can be defined. however, our results for the first multi-classification task were not reported due to an unknown error. we report our results for this task using the validation split. our proposal to solve this problem was grounded on knowledge integration to combine linguistic features and different kind of sentence embeddings. as commented in the introduction section, we wanted to evaluate a subset of language-independent linguistic features in a non-latin language. however, multilingual roberta separately outperformed slightly the results of combining different feature sets with ensemble learning or knowledge integration. as future work, we would like to extend the presented architecture by incorporating new feature extraction techniques to analyse their impact in precision. furthermore, we will focus on interpretability techniques. besides, regarding the application of emotions, we will evaluate the correlation of some linguistic features regarding anger and sadness with hate-speech in spanish with the datasets published at @xcite and @xcite .