Classifying Judgements using Transfer Learning (2020)

Abstract: We describe our method for classifying short texts into the APPRAISAL framework, work we conducted as part of the ALTA 2020 shared task. We tackled this problem using transfer learning. Our team, “orangutanV2” placed equal first in the shared task, with a mean F1-score of 0.1026 on the private data set.

## introduction systemic functional linguistics (sfl) is a theory of language which examines the relationship between language meaning and the functions in a social context @xcite . one popular framework that uses sfl is appraisal @xcite . the appraisal framework is based on the notion of uncovering the attitude of the author from the perspective of a potential listener or reader. it is used by linguists in analysing human behaviour from textual data @xcite @xcite . figure 1 shows an overview of the appraisal framework. the three main resources of the appraisal framework are; attitude, engagement and graduation @xcite . the at-titude framework is then subdivided into three subsystems; affect (emotions), appreciation (evaluation of natural and semiotic phenomena) and judgement (evaluation of people and their behaviour). the judgement subsystem can be divided into two categories: social esteem and social sanctions. social esteem primarily involves admiration and criticism and social sanction involves praise and condemnation. social sanctions can be further divided into three subcategories: normality (how usual one is), capacity (how capable one person is) and tenacity (how dependable one is). as for social sanc-tion it can be further divided into two subcategories; veracity (how truthful one is) and propriety (how virtuous one is). the robustness of the appraisal framework lies in its ability to be used in various different social contexts. it also offers linguists detailed strategies for realising the framework @xcite . since its debut, the appraisal framework has been widely used to explore how language is being used in various different environments such as in analysing examiners' reports on doctoral theses @xcite , donald trump's rhetoric tweets @xcite ), people's perception on the outcome of the brexit referendum @xcite and in teaching english as a second language (esl) @xcite . currently, linguists manually classify sentences using annotation software as there is no automated classification technique that exists to automate the task @xcite . thus, this problem sparked the interest of australasian technology association (alta) to organise a shared challenge task to develop a model that can automatically identify and classify human behaviour (judgement) ex-feature matched number (none) 104 normality 22 capacity 31 tenacity 21 veracity 2 propriety 33 multiple features 74 we present our participation in this challenge. we tackled this problem by utilising a pre-trained transfer learning model, albert @xcite , as a classifier. ## data set the data set foot_0 provided by the organisers is a collection of 300 tweets from semeval 2018 @xcite : 200 tweets for training and 100 for testing. the training data set consists of tweet id and the labelled annotations of subcategories of judgement which the tweet belongs to. if the tweet does not contain any subcategories it is marked as blank. we analysed the training data to understand the distribution and the patterns of category use. ## methodology first we handle class imbalances followed by preprocessing of our tweets. then, we perform unsupervised classification by utilising albert's pretrained model. we chose albert because it performs reasonably well on various different tasks such as offensive language detection @xcite , multiple-choice reading comprehen-sion @xcite , and question and answering @xcite . finally, we employed the cosine similarity measure in order to correct the mistakes made by our classifier. we have made our system's source code publicly available on github. foot_1 ## results kaggle was used as the platform for run submission. in kaggle, the test data provided to us by the alta organisers is split into public (public leaderboard) and private (private leaderboard). the private portion serves as a validation portion in order for the organisers to determine the effectiveness of all systems. the scores are evaluated using mean f 1. we summarise and present our results in ## conclusion and future work we presented our approach in order to automatically identify and classify judgement expressed in textual segments. we competed in the alta 2020 challenge under the team name of "orangutanv2" and placed equal first. our bestperforming system used a combination of transfer learning and document cosine similarity. despite setting a baseline for future work, we believe that there is still much work to be done in this area. as part of future work we are planning to tackle this problem in several ways, including: