Parsing Argumentative Structure in E nglish-as-Foreign-Language Essays (2021)

Abstract: This paper presents a study on parsing the argumentative structure in English-as-foreign-language (EFL) essays, which are inherently noisy. The parsing process consists of two steps, linking related sentences and then labelling their relations. We experiment with several deep learning architectures to address each task independently. In the sentence linking task, a biaffine model performed the best. In the relation labelling task, a fine-tuned BERT model performed the best. Two sentence encoders are employed, and we observed that non-fine-tuning models generally performed better when using Sentence-BERT as opposed to BERT encoder. We trained our models using two types of parallel texts: original noisy EFL essays and those improved by annotators, then evaluate them on the original essays. The experiment shows that an end-to-end in-domain system achieved an accuracy of .341. On the other hand, the cross-domain system achieved 94% performance of the in-domain system. This signals that well-written texts can also be useful to train argument mining system for noisy texts.

## introduction real-world texts are not always well-written, especially in the education area where students are still learning how to write effectively. it has been observed that student texts often require improvements at the discourse-level, e.g., in persuasiveness and content organisation aspects @xcite @xcite . worse still, texts written by non-native speakers are also less coherent, exhibit less lexical richness and more unnatural lexical choices and collocations @xcite @xcite . our long-term goal is to improve efl essays from the discourse perspective. one way to do this is by recommending a better arrangement of sentences, which enhances text coherence and comprehension @xcite . this may serve as feedback for students in the educational setting @xcite . the first step to achieve this goal, which is discussed in the current paper, is parsing argumentative structure in terms of dependencies between sentences. this is because the relationships between sentences are crucial to determine the proper order of sentences @xcite @xcite . this paper describes the application of argument mining (am) to efl essays. am is an emerging area in computational linguistics which aims to explain how argumentative discourse units (e.g., sentences, clauses) function and relate to each other in the discourse, forming an argument as a whole @xcite . am has broad applications in various areas, such as in the legal @xcite and news @xcite domains. also in the education domain, am is beneficial for many downstream tasks such as text assessment @xcite ), text improvement (as described above) and teaching @xcite . it is common in am to use well-written texts written by proficient authors, as do @xcite , @xcite , among others. however, there are more non-native speakers of english than native speakers in the world, and their writings are often noisy as previously described. yet, efl is a niche domain in am. this paper presents three contributions. first, this paper presents an application of am to efl essays. we parse the argumentative structure in two steps: (i) a sentence linking step where we identify related sentences that should be linked, forming a tree structure, and (ii) a relation labelling step, where we label the relationship between the sentences. several deep learning models were evaluated to address each step. we do not only evaluate the model performance based on individual links but also perform a structural analysis, giving more insights into the models' ability to learn different aspects of the argumentative structure. the second contribution is showing the effectiveness of well-written texts as training data for argumentative parsing of noisy texts. many am corpora exist for well-written texts because past studies typically assumed well-written input. corpora with noisy texts, such as the efl one we use here, exist, but are far more infrequent. in the past, well-written and noisy texts have been treated as two separate domains, and am systems were trained separately on each domain. we want to investigate how far the existing labelled corpora for well-written texts can also be useful for training parsers for noisy texts. to this end, we train parsers on both in-domain and out-of-domain texts and evaluate them on the in-domain task. for our out-of-domain texts, we use the improved versions of noisy efl texts. these improvements were produced by an expert annotator and have a quality closer to those of proficient authors. the third contribution of this paper is an evaluation of sentence-bert (sbert, @xcite ) in am as a downstream application setting. bert @xcite is a popular transformer-based language model (lm), but as it is designed to be fine-tuned, it can be suboptimal in low-resource settings. sbert tries to alleviate this problem by producing a more universal sentence embeddings, that can be used as they are in many tasks. the idea of training embeddings on the natural language inference (nli) task goes back to @xcite , and this is the sbert variant we use here. the nli task involves recognising textual entailment (te), and a te model has been previously used by @xcite for argumentation. we will quantify how the two encoders perform in our task. all resources of this paper are available on github. foot_0 ## related work argumentative structure analysis consists of two main steps @xcite . the first step is argumentative component identification (aci), which segments a text into argumentative discourse units (adus); then differentiates them into argumentative (acs) and non-argumentative components (non-acs). acs function argumentatively while non-acs do not, e.g., describing a personal episode in response to the given writing prompt. acs can be further classified according to their communicative roles, e.g., claim and premise. the second step is argumentative structure prediction, which contains two subtasks: (1) linking and (2) relation labelling. in the linking task, directed relations are established from source to target acs to form a structured representation of the text, often in the form of a tree. in the relation labelling task, we identify the relations that connect them, e.g., support and attack. in the education domain, argumentative structure interrelates with text quality, and it becomes one of the features that go into automatic essay scoring (aes) systems @xcite @xcite . end-to-end aes systems also exist, but hybrid models are preferred for both performance and explainability reasons @xcite . @xcite formulated am in three ways: as relation extraction, as sequence tagging and as dependency parsing. they performed end-toend am at token-level, executing all subtasks in am all at once. eger et al. achieved the highest performance in their experiments with the relation extraction model lstm-er @xcite . we instead use their sequence tagging formulation, which adapts the existing vanilla bidirectional long-short-term memory (bilstm) network @xcite , as it can be straightforwardly applied to our task. the dependency parsing formulation is also a straightforward adaptation as it models tree structures. the biaffine model is the current state-of-the-art of syntactic dependency parsing @xcite , and it has been adapted to relation detection and labelling tasks in am by @xcite . in a similar way, we also adapt the biaffine model to our argumentative structure. however, we use sentences instead of spans as adu, trees instead of graphs. most work in am uses well-written texts in the legal (e.g., @xcite and news (e.g., @xcite domains, but there are several am studies that concentrate on noisy texts. for example, @xcite focused on the aci task in web-discourse. @xcite investigated how to link arguments in discussion threads. in the education domain, @xcite studied the argumentation in persuasive essays. one of the prob-lems with the existing corpora is the unclear distinction between native and non-native speakers. additionally, to investigate and bridge the gap of performance between am systems on noisy and well-written texts, it is necessary to use a parallel corpus containing both versions of texts. however, none of the above studies did. ## dataset we use part of the "international corpus network of asian learners of english" @xcite , which we annotated with argumentative structure and sentence reordering ("icnale-as2r" corpus). 2 this corpus contains 434 essays written by college students in various asian countries. they are written in response to two prompts: (1) about banning smoking and (2) about students' part-time jobs. essays are scored in the range of @xcite . there are two novelties in this corpus: (1) it uses a new annotation scheme as described below and (2) contains a parallel version of essays which have been improved from the discourse perspective. therefore, this corpus can be used in many downstream tasks, e.g., employing argumentative structures for assessing and improving efl texts. it is also possible to extend the improved version of texts on other linguistic aspects. the corpus was annotated at sentence-level, i.e., a sentence corresponds to an adu. 3 in our annotation scheme, we first differentiate sentences as acs and non-acs, without further classification of ac roles. annotators then establish links from source to target acs, forming tree-structured representations of the texts. then, they identify the relations that connect acs. we instructed annotators to use the major claim, the statement that expresses the essay author's opinion at the highest level of abstraction, as the root of the structure. as there are no further classification of ac roles, the term "major claim" here refers to a concept, not an explicitly annotated category. as the last step, annotators rearrange sentences and performed text repair to improve the texts from a discourse perspective. there are four relations between acs: support (sup), attack (att), detail (det) and restate-ment (res). support and attack relations are common in am. they are used when the source sentence supports or attacks the argument in the target sentence @xcite . we use the detail relation in two cases. first, when the source presents additional details (further explanations, descriptions or elaborations) about the target sentence, and second, when the source introduces a topic of the discussion in a neutral way by providing general background. from the organisational perspective, the differentiation between detail and support is useful. while the source sentence in a support relation ideally follows its target, the detail relation has more flexibility. we also use a relation called re-statement for those situations where high-level parts of an argument are repeated or summarised for the second time, e.g., when the major claim is restated in the conclusion of the essay. detail and restatement links are not common in am; the first was introduced by @xcite and the second by @xcite , but both work on well-written texts. the combination of these four relations is unique in am. to improve the texts, annotators were asked to rearrange sentences so that it results in the most logically well-structured texts they can think of. this is the second annotation layer in our corpus. no particular reordering strategy was instructed. reordering, however, may cause irrelevant or incorrect referring and connective expressions @xcite . to correct these expressions, annotators were instructed to minimally repair the text where this is necessary to retain the original meaning of the sentence. for instance, they replaced pronouns with their referents, and removed or replaced inappropriate connectives. text repair is also necessary to achieve standalone major claims. for example, "i think so" with so referring to the writing prompt (underlined in what follows) can be rephrased as "i think smoking should be banned at all restaurants." figure 1 shows an example of our annotation scheme using a real efl essay. figure 2 then illustrates how the reordering operation produced an improved essay. the annotator recognised that (16) is the proper major claim of the essay in figure 1 . however, this essay is problematic because the major claim is not introduced at the beginning of the essay. thus, the annotator moved ( 16 ) to the beginning, and the whole essay is concluded by sentence sup (1) first of all, smoking is bad for your health. (13) in figure 2 . we also observe that crossing links happen in figure 1 , and they may suggest the jump of ideas, indicating coherence breaks. for example, sentence (14) describes nicotine and the annotator thinks that it properly connects to ( 9 ) which also talks about nicotine. therefore, it is more desirable to place sentences ( 9 ) and ( 14 ) close to each other, as shown in figure 2 . the reordered version is arguably better since it is more consistent with the argumentative-development-strategy prescribed in teaching, i.e., introduce a topic and state the author's stance on that topic, support the stance by presenting more detailed reasons, and finally concludes the essay at the end @xcite . we performed an inter-annotator agreement (iaa) study on 20 essays, using as annotators a phd student in english education (also an efl teacher -expert annotator) and the first author, both having a near-native english proficiency. we found the agreement to be cohen's κ=.66 for aci; κ=.53 for sentence linking; and κ=.61 for relation labelling @xcite . the sentence linking task sup (1) first of all, smoking is bad for your health. was the most difficult one, and it is understandable since a text may have multiple acceptable structures. the relation labels hardest to distinguish were between support and detail. this kind of annotation is expensive. also, there is no metric to measure the agreement on reordering. therefore, we chose to have the expert annotator annotate all essays in the production for the purpose of consistency. there are 6,021 sentences in the icnale-as2r corpus; 5,799 (96%) of these are acs and 222 (4%) are non-acs. an essay contains 14 sentences on average, and the average structural depth is 4.3 (counting the root at @xmath0. the corpus does not contain paragraph breaks. the most frequent relation label is support (3,029-56%), followed by detail (1,585-30%), attack (437-8%), and restate-ment (314-6%). in total, 105 out of 434 essays were rearranged (1-3 sentences were moved on average). as we have explained before, the expert annotator reordered a scattered set of sentences which logically form a sub-argument to be close in position to each other. they also placed the major claim at the beginning as opposed to the middle or the end of the essay. in general, the expert arranges the essays to be more consistent with the typical argumentative-development strategy prescribed in teaching. the text repair was done on 181 sentences, 123 (71%) of which attempt to repair the prompt-type error of the major claim. the remain-ing 58 sentences concern changes in connectives and referring expressions. ## parsing models we adopt a pipeline approach by using independent models for sentence linking, which includes the aci task, and relation labelling. although a pipeline system may fall prey to error propagation, for a new scheme and corpus, it can be advantageous to look at intermediate results. ## experimental results and discussion the dataset is split into 80% training set (347 essays , 4,841 sentences) and 20% testing set (87 essays, 1,180 sentences), stratified according to prompts, scores and country of origin of the efl learners. we are interested in how the am models trained on well-written texts may perform on more noisy texts. to find out, we train the models on both the original efl texts (in-domain) and the parallel improved texts (out-of-domain), then evaluated on the original efl texts. the difference between in-and out-of-domain data lies on the textual surface, i.e., sentence rearrangement, the use of connectives, referring expressions, and textual repair for major claims. since not all essays undergo any reordering, the out-of-domain data is roughly 75% the same as the in-domain data. the number of hidden units and learning rates (alongside other implementation notes) to train our models can be found in appendix a. we run the experiment for 20 times, 6 and report the average performance. the relation labelling models are trained and evaluated using sentence pairs according to the gold-standard. in the end-to-end evaluation (section 5.3), however, the input to the relation labelling model is the automatic prediction. statistical testing, whenever possible, is conducted using the student's t-test @xcite on the performance scores of the 20 runs, with a significance level of α = 0.05. ## conclusion this paper presents a study on parsing argumentative structure in the new domain of efl essays, which are noisy by nature. we used a pipelined neural approach, consisting of a sentence linking and a relation labelling module. experimental result shows that the biaffine model combined with the sbert encoder achieved the best overall performance in the sentence linking task (f1-macro of .323 on individual links). we also investigated mtl, which improved the sequence tagger model in certain aspects. in the sentence linking task, we observed that all models produced more meaningful structures when using sbert encoder, demonstrating its potential for downstream tasks. in the relation labelling task, non-fine tuning models also performed better when using sbert encoder. however, the best performance is achieved by a fine-tuned bert model at f1-macro of .595. we also evaluated our am parser on a crossdomain setting, where training is performed on both in-domain (noisy) and out-of-domain (cleaner) data, and evaluation is performed on the in-domain test data. we found that the best cross-domain system achieved 94% (acc of .321) of the in-domain system (acc of .341) in terms of end-to-end performance. this signals the potential to use wellwritten texts, together with noisy texts, to increase the size of am training data. the main challenge of argument parsing lies in the sentence linking task: the model seems to stumble when confronted with the hierarchical nature of arguments, and we will further tackle this problem in the future.