Automatically Predicting Judgement Dimensions of Human Behaviour (2020)

Abstract: This paper describes our submission to the ALTA-2020 shared task on assessing behaviour from short text, We evaluate the effectiveness of traditional machine learning and recent transformers pre-trained models. Our submission with the Roberta-large model and prediction threshold achieved first place on the private leaderboard.

## introduction language enables us to express evaluation of people, action, event, and things. this manifests as emotion and assessment of human behaviour and artefacts. the study of evaluative language has benefited from efforts in several disciplines such as linguistics, philosophy, psychology, cognitive science and computer science @xcite . in linguistics, the appraisal framework of @xcite provides a detailed classification scheme for understanding how evaluation is expressed and implied in language. in computer science, affective computing study evaluative language under the umbrella term of sentiment analysis with common tasks involving detection and classification of polarity and emotion, and aspectbased sentiment analysis, among others. sentiment analysis has benefited from the availability of usergenerated content on online platforms. the theory of appraisal proposed by @xcite has three categories of evaluative text: affect, judgement, and appreciation. these categories respectively model opinions in terms of emotions, norms, and aesthetics. utterances are viewed as indicating positive ("praising") or negative ("blaming") disposition towards some object (person, thing, action, situation, or event). the judgement dimensions are normality, capacity, tenacity, veracity, and propriety. each of the dimensions represents an answer to the following corresponding questions: the corpus used in this paper is annotated with the above judgement dimensions. @xcite automatically categorized appraisal into affect, judgement, and appreciation using a lexical approach that groups adjectives according to their semantic orientation. @xcite surveyed linguistic and computational approaches to the study of evaluative text. their analysis suggested that appraisal is a richer and more detailed task amenable to computational approaches subject to availability of data. they envision that appraisal analysis can contribute to the advances in affective computing. recently, @xcite showed that dimensions of appraisal can improve emotion detection in text. a similar observation was made by @xcite who found appraisal phrases as useful features for sentiment analysis. this paper investigates the capabilities of machine learning models in predicting the dimensions of human judgement expressed in short texts (tweets) as part of the alta-2020 shared task on assessing human behaviour @xcite . the task aims to advance computational techniques for analysing evaluative language. the use of neural networks has lead to significant performance improvements in nlp tasks. however, neural networks require a large amount of labeled data. on the contrary, the traditional machine learning models such as nbsvm are competitive in lowdata regimes @xcite and gelbukh, 2018). the recently introduced contextual representation learning models @xcite are pre-trained with language modeling objective on a large and diverse collection of text. the learned representation can be transferred to downstream tasks via fine tuning @xcite . we examine the effectiveness of using nbsvm and fine tuning a roberta-large model @xcite for predicting dimensions of judgement expressed in short text. ## methodology task. given a short text predict one or more judgement dimensions expressed in the given text. this is a multilabel classification problem where the labels consist of the five judgement dimensions. data. we employed the data provided by the organizers of the alta-2020 shared task @xcite . the training set has 198 tweets. each example is annotated with the presence or absence of each of the judgement dimensions as outlined in section 1. data pre-processing. we clean the text of each tweet by removing punctuation marks, digits, and repeated characters. we normalize urls and usernames (tokens that starts with the @ symbol). hashtags are converted to their constituent word(s) after removing the # symbol. ## conclusion we address the task of automatically predicting judgement dimensions in the context of the alta-2020 shared task. we evaluated the performance of a strong linear classifier, nbsvm with n-grams as features and a recent pre-trained language model, roberta-large. we observed that the nbsvm achieves our best score on the public leaderboard but it did not generalize to the private test set. the roberta-large model with decision thresholding strategy showed consistent performance on both the public and private leaderboards. with this model, we achieved the best overall score on the competition. while we achieved better performance with the roberta-large model, we think that the statistical power @xcite of the test set is limited due to the small sample size (100 examples). as such, it is difficult to differentiate performance improvement by chance from substantial model advantage. we hope to test our approaches on a larger test set in order to examine the robustness of our approaches.