Reproducibility and Automation of the Appraisal Taxonomy (2022)

Abstract: There is a lack of reproducibility in results from experiments that apply the Appraisal taxonomy. Appraisal is widely used by linguists to study how people judge things or people. Automating Appraisal could be beneficial for use cases such as moderating online comments. Past work in Appraisal annotation has been descriptive in nature and, the lack of publicly available data sets hinders the progress of automation. In this work, we are interested in two things; first, measuring the performance of automated approaches to Appraisal classification in the publicly available Australasian Language Technology Association (ALTA) Shared Task Challenge data set. Second, we are interested in reproducing the annotation of the ALTA data set. Four additional annotators, each with a different linguistics background, were employed to re-annotate the data set. Our results show a poor level of agreement at more detailed Appraisal categories (Fleiss Kappa = 0.059) and a fair level of agreement (Kappa = 0.372) at coarse-level categories. We find similar results when using automated approaches that are available publicly. Our empirical evidence suggests that at present, automating classification is practical only when considering coarse-level categories of the taxonomy.

## introduction with the rising popularity of social media platforms, such as twitter and facebook, we are experiencing an unprecedented surge of unstructured textual discourse ready to be analysed @xcite . supervised learning in natural language processing (nlp) has helped us extract the richness of the information found in these texts for purposes such as sentiment analysis @xcite , hate-speech detection @xcite , and question answering @xcite . the training task within supervised learning requires high-quality annotated data in order to perform well @xcite . supervised learning can apply theories of evaluative language @xcite . with highquality annotated data, we are confident that the task of identifying phrases of evaluative language can be automated. evaluative language allows us to analyse how we express our feelings, our assessments of people, situations and objects @xcite . as evaluative language is such a large and intricate discipline, herein we restrict ourselves to just the appraisal 1 (martin and white, 2003) taxonomy within it. appraisal gives linguists a systematic approach to evaluating language such as identifying and understanding how people make judgements about things (people and objects). the taxonomy has been widely used by linguists to analyse the language choices and attitudes used by writers in order to express their stances @xcite in various media such as in news biographies, examiners' reports and tweets @xcite @xcite . looking forward, appraisal could be used to automate the moderation of online comments @xcite ; in spite of automation, human moderators continue to be required for the final judgement call on some comments @xcite . the task of reading a comment (and analysing the language) is known to have a negative psychological impact on moderator's mental health @xcite . @xcite argued that the impact on human moderators could be lowered by reducing the number of comments they read. for example, if a comment is identified as having legal implications (based on appraisal analysis), the comment could be automatically rejected, or flagged for legal review. to date, much of the research in appraisal annotation has been descriptive in nature @xcite . @xcite proposed a structured approach known as the stepwise approach, however it stops short of providing guidelines to future researchers and has not been quantified. if there are to be robust data sets for studying appraisal then it is necessary to quantify the quality of the data sets already available, examine the practices involved in acquiring them, and strive to improve the techniques in a well grounded, measurable way. we focus the scope of this work on just the judgement subbranch of the appraisal taxonomy because there is publicly available data provided by the australasian language technology association (alta) for their 2020 shared task challenge @xcite . figure 1 shows the part of the appraisal taxonomy that focuses on judgement and where it fits in the hierarchy. in this paper, we investigate the following research questions: 1) the reproducibility and the reliability of humans annotating judgement sentences, and 2) the effectiveness of automated approaches to classify judgement. in order to answer our research questions, we first employed four annotators, each with a linguistics background, to re-annotate the alta data set. our experiments demonstrate significant levels of disagreement between the annotators at level 4 of the appraisal hierarchical taxonomy (fleiss κ = 0.089) as opposed to the more consistent results at level 2 of the taxonomy (κ = 0.558). we then compare the performance of three different systems submitted to the alta challenge. we observe a similar, relative effect: a κ score of 0.031 at level 4 and 0.206 at level 2. our qualitative analysis of the assessments shows that categorising the exact type of judge-ment can be difficult, as what constitutes morality (a part of judgement) is subjective-foresight and background context are required. we aim to encourage further research into the application of the appraisal taxonomy that is reproducible. this would collectively support our goal to build robust automated approaches to aid sfl practitioners in handling large datasets. to aid in future research, we have made our annotations appraisal affect (emotions felt) attitude judgement (people's behaviour) appreciations (reaction to objects/things) social esteem social sanction engagement graduation normality capacity tenacity propriety veracity level 4 level 3 level 2 level 1 level 0 figure 1: judgement branch of the appraisal hierarchical taxonomy (in context, and adapted from (stewart, 2015, p. 3).) and experimental data publicly available. foot_1 2 related work the appraisal taxonomy was originally developed by @xcite and it is used by linguists to study discourse across a diverse range of genres @xcite . appraisal captures the evaluative meaning (opinion) of the person who wrote a piece of text toward another person or an object. this taxonomy is widely used for many tasks including analysing biographies @xcite , donald trump's tweets @xcite , investigating online reviews of south park @xcite , advertising messaging @xcite and phd examiners' reports @xcite . there are five levels in the appraisal taxonomy (figure 1 ). some examples of tweets and where they fit within judgement are given in there is a substantial amount of work on building automated approaches to applying judge-ment classification to text @xcite @xcite @xcite . most of this work focuses on using a combination of machine learning approaches and a hand-built lexicon. for instance, @xcite , constructed lexicons from the seed words of @xcite and apply naive bayes and support vector machine (svm) classifiers to the task. they obtain an f 1 score of 0.345. one of the major drawbacks of these past approaches is that the appraisal terms in the lexicons were selected based on researchers' intuition, and it is unclear what these intuitions were. worse, the lexicons do not appear to be publicly available, thereby making it impossible to reproduce the experiments. the lack of publicly available appraisal data resulted in a stall in research, but recently the australasian language technology association (alta) organised a shared task and encouraged participants to build an automated system to identify the subclasses of judgement used in tweets @xcite . they made their data publicly available. 3 the results of the shared task were underwhelming, with the best system obtaining an f 1 score of 0.155 (aroyehun and gelbukh, 2020). in reaction, we sought to determine whether humans found this task equally difficult. prior work in this area by @xcite , used two annotators to annotate a book corpus and obtained an f 1 score of 0.434 at level 4 of the appraisal taxonomy, as opposed to an f 1 score of 0.532 at level 1 of the appraisal taxonomy. their data do not appear to be publicly available. @xcite discovered that tweets contain a higher proportion of judgement words than affect and appreciation words. judge-ment in tweets is especially insightful to an organisation interested in brand reputation, as an increasing number of consumers are using twitter to recommend brands to their friends @xcite . knowing how a product is being judged can bring insights to an organisation on how that product might be improved. collectively, these studies highlight the importance of evaluating the reliability of humans can perform the classification and compare the performance with existing approaches. ## human agreement level annotating judgement is difficult as sentences can contain varying degrees of objectivity and subjectivity. how these subjective sentences are finally classified can depend on the background of the annotator @xcite . to reduce subjectivity, we follow the recommendations of @xcite by annotating and reporting the scores at a tweet level (sentence level). we calculate the agreement levels based on the fleiss κ score and mean f 1 score. fleiss κ was chosen as it is appropriate when the given task is subjective @xcite , but recent work by @xcite recommends avoiding just κ when comparing the performance of automated approaches due to the kappa paradox @xcite . the kappa paradox arises because the κ statistic accounts for agreement by chance-which is low for machine learning approaches that closely follow the patterns in the training data. it is, thus, important to consider f 1 when comparing the performance in line with the findings of @xcite , our annotators' agreement scores drop as the appraisal classification moves from level 0 towards level 4. we explore some of the factors behind this in section 7. at level 4, if we were to include the golden set, g, as an annotator, there is a substantial level of disagreement among the annotators. similarly, at level 3, (social esteem versus social sanc-tions), we see that our annotators still have a low agreement. however, at level 0-2, we see moderate agreement between the annotators when the golden data set is not included, and a fair agreement level when the golden data set is included. the difference between f 1 and κ scores demonstrates the importance of reporting both scores. reporting f 1 alone gives the impression that this task is easy (has a high level of agreement) but the f 1 score does not take into account classifications that could have occurred by chance. the κ scores suggest that the task is difficult (has a low level of agreement) at level 4 of the taxonomy. this sheds light as to why the automated approaches that were built by the participants of the alta task performed poorly @xcite , as we find even humans disagree at levels 3 and 4. ## classifiers' agreement level in this section, we discuss our experiments for measuring the effectiveness of automated approaches. we consider all three systems that were used to submit runs to the alta 2020 shared task because the source code to these is publicly available, and we were able to gain access to the submissions. we did not compare to earlier systems as neither they, nor their word lists, nor the data they trained on are publicly available. the systems we used are as follows: mean f 1 scores of each tweet against g are reported in this is hardly surprising, since the systems were trained on a limited amount of data, but the annotators were able to draw from years of experience. nonetheless, nlp-cic and orangutanv2 are able to distinguish judgement from non-judgement fairly accurately at level 0-2. nits may be failing due to the use of xgboost which is susceptible to outliers, and certainly produces results that differ from the transformer-based models of nlp-cic and orangutanv2. ## performance of humans and classifiers we then proceed to compare the performance of human annotators with the automated approaches. we are interested in knowing at which level of the appraisal taxonomy is there a significant difference between the performance of a human and of a machine. we picked the best performing human annotator who obtained the highest f 1 score against the golden data set at every level to be evaluated (c). we determined that scores of c for each tweet are not normally distributed by running the shapiro-wilk test @xcite and obtaining a p-value of 0.0466 (not normally distributed at the significance level of @xmath0.05). knowing that the data are not normally distributed we chose the wilcoxon signed-rank test @xcite to test for significant differences between c and each automated system. we ran the test after removing ties, which minimises type one error @xcite . level nlp-cic orangutanv2 nits 0-2 0.198 0.221 0.314 3 0.102 0.113 0.214 4 0.038 * 0.042 * 0.095 we present our results in so, as our effect size (@xmath2.15) is very small. we report our fleiss κ scores in ## qualitative analysis observing the high level of disagreement among annotators and g (mean κ at level 4 being 0.199), we examined each disagreement in turn. they all fall into two categories: consider the following category 1 example: annotators a, c and d marked this normality while a and b marked it capacity (a marked it twice). the golden assessor, g, marked it none. when we apply the appraisal taxonomy manually, we observed that the first part of the sentence "absolutely love @unqualified but can't listen to it during my commute on the subway" being affect, however in the second part "because i burst out laughing and people stare" is judgement. we believe this sentence does contain judge-ment. in many societies, laughing loudly on a train is considered rude, as is listening to music loudly. we agree with the majority of our annotators that this warrants classification as normal-ity, and if we follow the methodology of @xcite , this tweet falls into both categories. all of the automated approaches also predict normal-ity. our analysis shows three instances of category 1. although, from our visual inspection, these sentences contain judgement, we are not sure why g chose none (and we have no way to investigate this). as an example of category 2: "so disappointed in myself for spending £50 on an outfit for meeting a boy #noselfcontrol #nervous ." annotator g tagged this normality, disagreeing with all the other annotators who marked it ca-pacity. there are two ways that the sentence can be interpreted, for our annotators, they view the person as not being capable of controlling their impulses to purchase an outfit for their date. as for g, the annotators plausibly view it as normality as it was normal for people to be in their best outfit and behaviour when they are out on a date, thus it was normal to spend money on a dress. we believe that both of these categories are correct, especially as there are no well-defined criteria to distinguish between capacity and normality. to address this issue, we suggest clarifying annotation guidelines given to the assessors, and the criteria used to distinguish the different judgement categories. all of the automated approaches predicted pro-priety. this is likely to be due to the lack of data-there is only one example in the training data set that has a sentence similar to the sentence above: "incredibly shocked and disappointed with @united customer service. really making me rethink flying with them in the future. #unhappy". that sentence is marked as propriety. our findings reflect the findings of tayyar @xcite , who have demonstrated that bert fails to generalise properly when training and test data are significantly dissimilar even though these data sets are very similar in nature. there are four instances of category 2 disagreement. another explanation for discrepancies between our annotators and g is that annotating tweets is tricky because the tweeter and the annotator are subject to different cultural and personal views of what judgement is. the data we use consists of tweets from different individuals and we have different individuals assessing them, thus we can expect high levels of disagreement. by comparison, the work of @xcite , applies appraisal theory to tweets from one individual (donald trump), and so the assessors are able to better understand the message behind the tweet. this could be the reason why much of the previous work on appraisal has focused solely on a particular topic or person and not a generalised situation. one way to address discrepancy among annotators is for the annotators to meet and discuss differences-and to improve the assessment guidelines by writing clear-cut criteria for distinguishing difficult cases. we hope to see more diversified data sets released in the future so that we can validate the generalisability of automated judgement approaches. ## recommendations & limitations our analyses show that: 1) annotating level 4 of judgement categories is challenging as there is ambiguity in the interpretation of the text; and 2) the evaluation and the reasoning presented in the appraisal literature are very rarely complete. the latter can be addressed by making the data sets publicly available and also sharing the assumptions and annotation guidelines. it is paramount to have these guidelines as it helps reproduce research results. moreover, sharing these rules would make it easier to automate approaches that classify ap-praisal-bearing sentences. by addressing these gaps, we believe that annotating level 4 of ap-praisal would be clearer, although how the rules differ or are similar from one data set to another is yet to be seen. our work has limitations. our data set is small and focuses solely on twitter and the judgement branch of appraisal. we find that performing discourse analysis can be challenging especially when it is related to judging the morality of a person from a single tweet. this can surface different perspectives based on different assumptions based on a tiny piece of text @xcite . we evaluated our automated approaches using systems built for the alta shared task. all of these models were based on bert and we believe that these models can be further improved by finetuning, which has been shown to improve performance elsewhere @xcite . another plausible cause of the poor performance of these models is that the data set that was trained on and evaluated on was small, and thus was likely to contain bias. as a direction for future work, it would be inter-esting to revisit some of the techniques from earlier work mentioned in section 2. of course, the lexicons would need to be reproduced as closely as possible, but we hypothesise that the deep learning models could be further improved by using such techniques. ## conclusions in this study, we investigated two topics: (1) reproducibility and the reliability of a popular sfl taxonomy, appraisal, focusing on judgement annotation, and (2) the effectiveness of automated approaches to assessing judgement. to carry out our investigation in a systematic manner, we employed four linguists to carefully re-annotate the publicly available alta 2020 shared task data set and used three publicly available, automated approaches. we then performed experiments quantifying and evaluating the performance of our annotators and automated approaches. we find a low level of agreement when annotating judgement despite using annotators with linguistics backgrounds. we obtained a fleiss κ score of 0.059 when using level 4 (most detailed categories) within the appraisal's hierarchical taxonomy as opposed to 0.372 when using level 2 (coarse-grained categories). we find a similar pattern with the automated approaches. we obtain a fleiss κ score of 0.031 at level 4 and 0.206 at level 2 of the taxonomy. although the low κ score in automated approaches is attributed to the nature of κ statistical penalising agreement not occurring by chance, our f 1 score (0.605 at level 2 and 0.155 at level 4) supports our earlier findings that humans find classifying judgement to be difficult. furthermore, we find that there is no statistical significance between the performance of our best performing annotators and of the best performing system when working with level 2 category of the appraisal taxonomy, thus we argue that automation of judgement is possible at this level as automated systems are already performing at human levels. our analyses sheds light on the challenges in reproducibility of appraisal annotation. we believe that the poor scores of human annotators and automated approaches are due to a multitude of factors including the lack of publicly available data sets (examples), the absence of details such as prior assumptions made by the annotators, and the lack of generally available clear and concise annotation guidelines. we have publicly released our data and analysis to encourage more research into appraisal. we believe that the application of appraisal to tweets and other discourse will enable the application of appraisal to other domains (such as ecommerce).