Multi-task and Multi-corpora Training Strategies to Enhance Argumentative Sentence Linking Performance (2021)

Abstract: Argumentative structure prediction aims to establish links between textual units and label the relationship between them, forming a structured representation for a given input text. The former task, linking, has been identified by earlier works as particularly challenging, as it requires finding the most appropriate structure out of a very large search space of possible link combinations. In this paper, we improve a state-of-the-art linking model by using multi-task and multi-corpora training strategies. Our auxiliary tasks help the model to learn the role of each sentence in the argumentative structure. Combining multi-corpora training with a selective sampling strategy increases the training data size while ensuring that the model still learns the desired target distribution well. Experiments on essays written by English-as-a-foreign-language learners show that both strategies significantly improve the model’s performance; for instance, we observe a 15.8% increase in the F1-macro for individual link predictions.

## introduction argument mining (am) is an emerging area that addresses the automatic analysis of argumentation. many recent studies commonly try to tackle two major tasks @xcite . the first of these is argumentative component identification, in which argumentative units (acs) and nonargumentative components (non-acs) including their boundaries are determined. acs can be further classified according to their role in argumentation, e.g., major claim, foot_0 claim and premise @xcite . the second task is called argumentative structure prediction, which first establishes links from source to target acs (this is called the linking task) and then labels the relationship between them, for instance using the support and attack relation labels (this is called the relation labelling task) @xcite . the linking task has been identified by earlier works as particularly challenging @xcite @xcite . there are many possible combinations of links between textual units, and a linking model has to find the most proper structure out of a very large search space. another typical challenge in am is the size of annotated corpora @xcite . corpus construction is a complex and time-consuming process; it also often requires a team of expert annotators. existing corpora in am are relatively "small" compared with more established fields, such as machine translation or document classification. this hinders training am models when using a supervised machine learning framework. in this paper, we perform the linking task for essays written by english-as-a-foreign-language (efl) learners. given an essay, we identify links between sentences, forming a tree-structured representation of argumentation in the text. figure 1 illustrates the task. our contributions are twofold. first, we propose two structural-modelling related auxiliary tasks to train our model in a multi-task learning (mtl) fashion. past studies commonly executed several am tasks or using discourse and rhetorical auxiliary tasks in the mtl setting (cf. section 2). in comparison with other works, our auxiliary tasks are advantageous in that they do not require any additional annotation. second, we combine multi-corpora training with a selective sampling strategy to increase the training data size. we explore the use of mixed training data of in-domain efl and out-domain foot_1 student-essay corpora that were annotated using different schemes and of different quality, while ensuring that the model still learns the properties of the target in-domain data well. the choice of efl texts in this study aims to contribute to a less attended area. in am, it is common to use well-written texts by proficient authors (e.g., @xcite . however, student texts often suffer from many problems because they are still in the learning phase. even more, efl texts are also less coherent and less lexically rich, and exhibit less natural lexical choices and collocations @xcite . there are more non-native english speakers @xcite , and yet, to the best of our knowledge, only one preceding study in am concerned the efl genre @xcite . the codes accompanying this paper are publicly available. foot_2 2 related work ## dataset our target texts are sourced from the icnale-as2r, a corpus of 434 essays written by asian college students with intermediate proficiency. 4 there are 6,021 sentences in total with 13.9 sentences on average per essay. to the best of our knowledge, this is the only currently publicly-available am corpus focusing on efl texts. the corpus is annotated at sentence-level, that is, a sentence corresponds to one argumentative unit. the corpus differentiates sentences as acs (5,799 sents.) and non-acs (222 sents.), without further classification of ac types. links are established between acs to form tree structure (avg. depth of 4.3, root at depth 0 in the corpus), where the major claim acts as the root. four relation labels are employed to label the links: support, attack, detail and restatement. ## proposed method in this work, we experiment on the linking task. given an entire essay of n sentences as input, s 1 , ..., s n , a linking model outputs the distance d 1 , ..., d n between each sentence s i to its target. for instance, if a sentence is connected to its preceding sentence, the distance is @xmath0. we consider those sentences that have no explicitly annotated outgoing links as linked to themselves (@xmath1; this concerns major claims (roots) and non-acs. we do not consider labelling the links with their relationships in this paper. figure 2 shows the distance distribution between the source and target sentences in the icnale-as2r corpus, ranging [-26, ..., +15]. adjacent links predominate (50.4%). short distance links (2 ≤ |d| ≤ 4) make up 21.2% of the total. backward long distance links at @xmath2.6%, whereas forward long distance links are rare (1.0%). self-loops make up 10.9% of the total. following recent advances in am, we approach the linking task as a sentence-to-sentence dependency parsing. ## structural metric in this paper, we do not evaluate the model performance only on individual link predictions but also analyse the structural properties of the outputs, giving more insights into the models' ability to learn different aspects of the argumentative structure. we also investigate whether our proposed strategies improve the model performance concerning the grouping of related sentences into the same sub-tree, a challenge to linking models previously (manually) observed in putra et al. (2021b) (cf. section 2.1). a metric is needed to quantify the improvement on this aspect, and we propose to use a novel structural metric called mar dset , that was developed by putra et al. (2021a). 5 given two structures a and b (i.e., predicted and gold structures), we can quantify their similarity based on the presence of common substructures. we first define a descendant set (dset) of a node x as the set consisting the node x itself and its descendants. figure 6 shows examples of dsets (brackets given below the node id). 6 for example, the dset of node-2 in annotation a of figure 6 is {2, 3, 4, 5}. two corresponding nodes in a and b are required to have identical dsets in order to score a value of 1. for example, the matching score for node-2 between annotation a and b in figure 6 is 0, while the matching score for node-3 is 1. non-ac nodes are counted as a match if they are deemed non-argumentative in both structures. with this formulation, we can get a vector v representing the matching scores for all nodes in the two structures. for example, @xmath3, 0, 1, 1, 0] for figure 6 . the similarity score between two structures are then calculated as mar @xmath4, 5 mar stands for mean agreement in recall. descendant set (dset) denotes the unit of analysis. 6 in the implementation, we use sentence position in the input text as node id. ## conclusion in this paper, we presented a study on the argumentative linking task; given an essay as input, a linking model outputs the argumentative structure in a tree representation. we conducted experiments using an efl corpus, namely icnale-as2r. we extended the state-of-the-art biaffine attention model using a novel set of structural auxiliary tasks in the multi-task learning setup. additionally, we also proposed a multi-corpora training strategy using the pec to increase training instances. it has to be noted that simply increasing the training data size does not guarantee improvement. we need to ensure that the model still learns the desired target distribution as well. to this end, we filtered pec essays using a selective sampling technique. these two strategies provided useful supervision signals to the biaffine model and significantly improved its performance. the f1-macro for individual link predictions was boosted to .374 from .323. our strategies also improved the model performance on the structural aspects, achieving the mar dset of .452 from .419, and the f1-macro of .622 from .601 for the qact task. a possible future direction is to evaluate whether our proposed methods are beneficial for more complex discourses, such as scientific articles. it is also necessary to provide an even richer supervision signal to improve the linking performance. one possibility is to pre-train the model on the anaphora resolution task. yuxiao ye and simone teufel. 2021. end-to-end argument mining as biaffine dependency parsing. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 669-678, online. association for computational linguistics.