Overview of the 2020 ALTA Shared Task: Assess Human Behaviour (2020)

Abstract: The 2020 ALTA shared task is the 11th in stance of a series of shared tasks organised by ALTA since 2010. The task is to classify texts posted in social media according to human judgements expressed in them. The data used for this task is a subset of SemEval 2018 AIT DISC, which has been annotated by domain experts for this task. In this paper we introduce the task, describe the data and present the results of participating systems.

## introduction human behaviour can be negatively or positively assessed based on a reference set of social norms. when judgement is explicitly stated in narratives, e.g., "they are hard-working and honest.", we can attempt to encounter appraisal words such as "hardworking" and "honest" used between interlocutors for advancing their judgement. attitude positioning plays an important role in martin and white's (2005) appraisal framework 1 (af) for analysing someone's use of evaluative language to negotiate solidarity. to the best of our knowledge, no prior work has attempted to automatically codify text using the af judgement categories. the goal of the 2020 alta shared task is to develop a computational model that can identify and classify judgements expressed in textual segments. participants are challenged to predict the judgement appraised by classifying each short-text message into one or more label candidates (or none): normality, capacity, tenacity, veracity, propriety. ## the 2020 alta shared task the 2020 alta shared task is the 11th of the shared tasks organised by the australasian lan-1 https://www.grammatics.com/appraisal/ guage technology association (alta). as in previous shared tasks, it targets university students with programming experience, but it is also open to graduates and professionals. the general objective of these shared tasks is to introduce interested people to the sort of problems that are the subject of active research in a field of natural language processing. depending on the availability of data, the tasks have ranged from classic but challenging tasks to tasks linked to very hot topics of research. details of the 2020 alta shared task and past tasks can be found in the 2020 alta shared task website. @xcite there are no limitations on the size of the teams or the means that they may use to solve the problem. we provide training data but participants are free to use additional data and resources. the only constraint in the approach is that the processing must be fully automatic -there should be no human intervention. as in past alta shared tasks, there are two categories: a student category and an open category. the prize is awarded to the team that performs best on the private test set -a subset of the evaluation data for which participant scores are only revealed at the end of the evaluation period. ## the appraisal framework the appraisal framework (af) is concerned with the use of linguistic markers for identifying and track the ways attitudes are invoked in authored text. the framework defines three subsystems for evaluative meaning making (1) attitude; (2) engagement; and (3) graduation. each of these are further divided in to other subsystems (figure 1). in particular, the attitude framework is divided into three subsystems: (1) affect (registering of emotions); (2) appreciation (evaluations of natural and semiotic phenomena); and (3) judgement (evaluations of people and their behaviour). the judgement subsystem has two regions: social esteem and social sanction. the subcategories of each of these two regions form the target labels for the 2020 alta shared task. in particular: social esteem tends to function as admiration or criticism and can be subdivided into three subcategories: normality (how unusual one is): "he is oldfashioned". capacity (how capable one is): "self-driven 12 year old is a maths genius". tenacity (how resolute one is): "they are hardworking and honest". social sanction functions as praise or condemnation and can be subdivided into two subcategories: veracity (how honest/truthful one is): "they are hard-working and honest". propriety (how ethical one is): "she is too arrogant to learn the error of her ways". the judgement system is used to assess human behaviour and their position on certain social norms. further details and examples can be found in the appraisal website. @xcite 4 data the source data of the 2020 alta shared task is a subset of the semeval 2018 ait disc dataset. @xcite a total of 300 tweets have been manually annotated in a two-stage process. the annotation was first annotated by two linguists from two australian universities (university of wollongong and university of new south wales) and then double-checked by two other linguists from the same two universities. the data were subsequently split into a training set of 200 tweets, and a test set of 100 tweets. each tweet was annotated with one or more (or none) of the following labels: normality, capacity, tenacity, veracity, propriety. ## evaluation as in previous alta shared tasks, the task was managed as a kaggle in class competition. this year's task name was "alta 2020 challenge". @xcite the kaggle-in-class platform enabled the participants to download the data, submit their runs, and observe the results of their submissions in a leaderboard instantly. as is common in kaggle competitions, when a participant team submits their results, the public leaderboard shows the evaluation results of part of the test data, and the results of the remaining test data are held for the final ranking. by following the public leaderboard, a team can then gauge the performance of their system in comparison with that of other systems in the same public test set. a team can choose up to two of their runs for the final ranking. if a team chooses runs for the final ranking, the best results on these runs on the private partition of the test data will be used. if a team does not choose any runs, the private evaluation results of the run with the best results on the public partition will be chosen. the systems were evaluated using the mean of the f1 score over the test samples (1), ## participating systems in total 5 teams registered for the competitions, all of them in the student category. of these, 3 teams submitted runs. team nlp-cic experimented with logistic regression and roberta (aroyehun and gelbukh, 2020). whereas the logistic regression classifier obtained the best results in the public leaderboard, it performed much worse in the private leaderboard. in contrast, the roberta classifier obtained consistent results in both the public and private leaderboards. team orangutanv2 designed classifiers using albert and transfer learning (parameswaran et al., 2020). after observing that 22 tweets from the test set are also in the training set, they also incorporated a component that performed cosine similarity with the samples from the training data. team nits experimented with ensemble approaches (khilji et al., 2020). they obtained pretrained word embeddings and incorporated polynomial features. these features were fed to decision tree and extreme gradient boosting (xgboost) classifiers. ## results the results indicate that this task has been particularly challenging and there is room for improvement. a possible reason for the difficulty of this task is the small number (200) of annotated samples available. another reason for the low results is the relatively large percentage of samples with empty judgements. in particular, 60% of the test data had empty judgements. according to formula (1), the f1 score of test samples with no annotations is 0. this means that the upper bound with this test data is 0.4. ## conclusions the aim of the 2020 alta shared task was to predict the judgement of short texts according to martin and white's (2005) appraisal framework. the task proved challenging, presumably due to the small amount of annotated data and the sparse annotations in the data.