Span Identification of Epistemic Stance-Taking in Academic Written E nglish (2023)

Abstract: Responding to the increasing need for automated writing evaluation (AWE) systems to assess language use beyond lexis and grammar (Burstein et al., 2016), we introduce a new approach to identify rhetorical features of stance in academic English writing. Drawing on the discourse-analytic framework of engagement in the Appraisal analysis (Martin & White, 2005), we manually annotated 4,688 sentences (126,411 tokens) for eight rhetorical stance categories (e.g., PROCLAIM, ATTRIBUTION) and additional discourse elements. We then report an experiment to train machine learning models to identify and categorize the spans of these stance expressions. The best-performing model (RoBERTa + LSTM) achieved macro-averaged F1 of .7208 in the span identification of stance-taking expressions, slightly outperforming the intercoder reliability estimates before adjudication (F1 = .6629).

## introduction automated writing evaluation (awe) systems make it possible to assess students' writings and provide useful feedback efficiently @xcite . from the language assessment perspective, however, usefulness is multifaceted (e.g., @xcite and, in many parts, depends on what areas of writing ability a given system can measure and give feedback on @xcite . while many awe systems to date focus on lexical, syntactic, organizational, and topical aspects of students' writing (e.g., @xcite , the construct of writing (i.e., writing skill) is known to be far more complex and includes pragmatic and rhetorical knowledge @xcite . accordingly, recent studies have included constructs such as discourse moves and steps (e.g., @xcite , source use and citations @xcite , and argument structures using rhetorical structure theory @xcite . given the increasing focus on the assessment of the ability to construct effective persuasive texts @xcite , innovative use of nlp is needed how to assess these social and rhetorical constructs of writing (e.g., @xcite @xcite . ## engagement discourse treebank (edt) the edt currently comprises 4,688 sentences with manually annotated engagement resource spans (126,411 tokens; 11,856 spans), which were sampled from corpora of academic english or closely related genres (see definition of in-domain text below). the version of edt used to train the machine learning models presented in this paper is accessible at https://github.com/lcr-adslab/engagement-discourse-treebank. the most recent version of the annotation guideline is accessible through the following github page: https://egumasa.github.io/engagement-annotationproject/. ## annotation procedure the annotation team consisted of two primary annotators (undergraduate students; linguistics majors) and the principal investigator (pi) (the first author, who was a ph.d. candidate in a functional linguistics program and holds a master's degree in second language acquisition and english language teaching). the annotation project comprises the following four steps-annotator training (section 4.1), iterative consensus building (section 4.2), independent annotation (section 4.3), and doublechecking and quality assurance (section 4.4). the annotation comprised two tasks-detecting spans and assigning one functional label for each span. ## model architectures the identification and classification of engagement strategies were formulated as a span identification task (e.g., @xcite . our proposed architectures most closely resemble the span enumeration approach in @xcite , where candidate spans are generated greedily (using n-grams and dependency subtrees). figure 2 shows three variants of our neural architecture. we started from the baseline spacy span categorizer model @xcite . we then gradually built the model complexity, guided by previous work in span identification (e.g., @xcite @xcite and our intuitions as linguists. the basic span categorizer pipeline consists of token embedder, span suggester, and span categorizer. ## methods we implemented the three architectures through spacy version 3.4 @xcite . all models were trained on a quad nvidia tesla k80 gpu with 12gb ram. all models were optimized with adam optimizer. ## results ## discussion the results of the 5-fold cvs indicated that the proposed architectures performed as well as (or even outperformed) the inter-annotator agreement baseline set for the study. the results also suggested that our roberta + lstm and dual-roberta models tended to perform better than the spacy default spancat model @xcite . it is noteworthy that the additional bi-lstm layer appeared to enhance the stability of the model. although the use of a bi-lstm layer on top of a transformer encoder is not uncommon in span identification tasks, its reported benefits have been mixed @xcite @xcite . the gain in this study can be explained in two ways-additional sequential information and dimensional reduction. in a simple explanation, the architecture benefited from the additional sequence information provided by bi-lstm. at least one previous study @xcite reported similar gains in additional lstm layer, particularly when the span suggestion components were similar to the current greedy approach. thus, it could be that the additional lstm helped to refine the embedding for this particular span enumeration architecture @xcite . in addition to this explanation, it is also possible that lstm worked as a dimension reducer (while maintaining direct sequential information). future research may clarify the potential reasons for this stability in the span identification task (which is out of the scope of the current study). apart from the machine learning experiment, our inter-annotator agreement showed that the span annotation of engagement resource may be a challenging task, particularly for undergraduate annotators (linguistics majors) who were trained over 10 weeks. however, our annotator agreement substantially improved upon the previously published benchmark by @xcite . the moderate reliability in this study may provide further evidence to @xcite claim regarding the lack of explicit guidelines and methodological discussions pertaining to the identification of engagement resources in discourse samples. thus, it is hoped that the present annotation guideline may serve as a resource to guide future methodological improvement in discourse annotation of engagement resource analysis (see @xcite . ## conclusion in this paper, we reported a new approach to identifying stance-taking expressions in english texts in academic domains. specifically, we introduced a new human-annotated corpus of academic english that draws on a discourseanalytic framework of the engagement system from the appraisal framework @xcite . we also reported an end-to-end system that can conduct automated span identification of stance-taking strategies based on the engagement framework. the experimental result indicates that the system can outperform inter-annotator reliability estimates by a 5-6% gain in the macroaveraged f1 score. the finding, although preliminary, opens a new avenue for feature engineering for the next-generation awe systems @xcite , expanding the constructs measured by the awe engines. a follow-up study by the first author shows that the engagement features can explain the writing scores above and beyond the existing linguistic features at the levels of lexis, grammar, and cohesion @xcite . while end-to-end score prediction models may be used to obtain accurate score predictions, the features introduced in this paper may be used in conjunction with such end-to-end scoring engines to maintain the explainability and interpretability of the scores. the visualization of stance-taking features (see figure 1 and demo) can also be presented to learners to highlight the patterns of stance-taking in both model and student essays.