APE : Argument Pair Extraction from Peer Review and Rebuttal via Multi-task Learning (2020)

Abstract: Peer review and rebuttal, with rich interactions and argumentative discussions in between, are naturally a good resource to mine arguments. However, few works study both of them simultaneously. In this paper, we introduce a new argument pair extraction (APE) task on peer review and rebuttal in order to study the contents, the structure and the connections between them. We prepare a challenging dataset that contains 4,764 fully annotated review-rebuttal passage pairs from an open review platform to facilitate the study of this task. To automatically detect argumentative propositions and extract argument pairs from this corpus, we cast it as the combination of a sequence labeling task and a text relation classification task. Thus, we propose a multitask learning framework based on hierarchical LSTM networks. Extensive experiments and analysis demonstrate the effectiveness of our multi-task framework, and also show the challenges of the new task as well as motivate future research directions.

## introduction argument mining is an important research field that attracts growing attention in recent years @xcite . it is applied in real-world applications such as legal documents @xcite , online debates @xcite , persuasive essays @xcite , etc. most works in argument mining field focus on modeling arguments in monologues. those focusing on the detection of agreement and disagreement in online interactions review: [the authors introduce an extension of continuous ranked probability scores (crps) to the time-to-event setting termed survival-crps for both right censored and intervalcensored event data.]non-argu ... [the claim that the proposed approach constitutes the first time a scoring rule other than maximum likelihood seems too strong, unnecessary and irrelevant to the value of the presented work.]review-1 [it is not clear how did the authors handle the irregularity (in time) of ehr encounters in the context of an rnn specification. also, if the rnn specification considered is similar to martinsson, 2016, why this wasn't considered as a competing model in the experiments?]review-2 ... ## related work argument mining. there is an increasing number of works in the computational argumentation research field in recent years, such as argument mining @xcite , argument relation detection @xcite , argument quality assessment @xcite @xcite , argument generation @xcite @xcite , etc. @xcite and @xcite both propose pipeline approaches to identify argumentative discourse structures in persuasive essays, which mainly includes two steps: extracting argument components and identifying relations. in terms of the task and the dataset, we intend to extract argument pairs from two passages simultaneously, while they focus on a single passage. in addition, we present a multi-task framework to tackle our proposed task instead of using a pipeline approach. @xcite aim to extract arguments from a large corpus of posts from online forums. they frame their problem as two separate subtasks: argument extraction and argument facet similarity. chakrabarty et al. (2019) focus on argument mining in online persuasive discussion forums based on the cmv dataset. compared to both of their datasets, our dataset's contents are more academic and are strongly inclined to attack the other argument. scale-wise, our dataset consists of 156k sentences, while chakrabarty et al. (2019)'s dataset has 2,756 sentences. peer reviews mining. previous works focus on studying peer reviews by introducing new datasets or understanding the structure and the content of peer reviews. kang et al. (2018) present a dataset named peerread collected from venues including acl, nips and iclr. it consists of 14.7k paper drafts with their corresponding accept/reject decisions and 10.7k peer reviews for a subset of papers. they also propose several tasks based on their dataset, such as predicting whether a paper is accepted. hua et al. (2019b) then present a partially annotated dataset named ampere collected from iclr, uai, acl and neurips. they use their dataset to detect the propositions and to classify them into different types for understanding the contents of peer reviews. compared to the two works described above, our work has two main differences. first, instead of only looking at peer reviews, we propose a task to study the reviews and rebuttals jointly. another difference lies in dataset: ours is fully annotated with 4,764 pairs of reviews and rebuttals, while only 400 reviews are annotated in ampere. as mentioned, few works touch on the rebuttal part, which is the other important element of the peer review process. gao et al. (2019) ## rr dataset we create the review-rebuttal (rr) dataset to facilitate the study of argument pair extraction. to the best of our knowledge, this is the first dataset for argument mining between paired reviews and rebuttals. we first describe the process of data collection, and then present the annotation details. ## task description in this work, we intend to automatically detect the argument pairs from each passage pair of review and rebuttal. specifically, we first perform argument mining on the two passages, a review passage of m sentences r each mined argument consists of one or more sentences from the passage with no overlapping with other arguments, such as review-1, review-2, reply-1, and reply-2 in ## multi-task learning based extraction as shown in figure 1 , we propose a multi-task learning framework with hierarchical lstm networks, named mt-h-lstm-crf, to tackle the two subtasks, namely, argument mining and sentence pairing. the red dotted box on the left shows our sentence encoder which uses the pre-trained bert @xcite token embeddings as the input of a token-level bilstm (t-lstm) to get sentence embeddings. then, the encoded sentence embeddings are input to the second layer of lstm on the sentence level (s-lstm) to encode the sequence information of the passages. finally, two types of labels, i.e., iobes and 1/0, are predicted simultaneously from the shared features generated from the s-lstm. note that the aforementioned procedure is of multi-task training. during the inference, the trained multi-task model will be decoupled into two sub-modules to perform the two subtasks in a pipeline manner to extract the final argument pairs. ## experiments 6.1 data preparation we first split our rr dataset on review-rebuttal passage-pair level randomly by a ratio of 8:1:1 for training, development and testing, namely rrpassage. in rr-passage dataset, all argument pairs from the same passage pair are put into only one of the training, development and testing sets. however, different review-rebuttal passage pairs of the same submission could be put into different sets. since different reviewers may discuss similar issues for one submission, different review-rebuttal passage pairs of the same submission may share similar context information. to alleviate this effect, we also prepare another dataset version split on the submission level, namely rr-submission. it also follows the ratio of 8:1:1 for training, development and testing tests. in rr-submission, multiple review-rebuttal passage pairs of the same submission are in the same set. ## conclusions and future work in this paper, we introduce a new task of extracting argument pairs from review and rebuttal passages, which explores a new domain for the argument mining research field. a new large-scale and challenging dataset rr is collected and fully annotated to facilitate the study of the proposed task. we then propose a multi-task learning approach based on hierarchical lstm networks to work towards this problem. in the future, we will explore the latent information between peer reviews and author responses to improve argument pair extraction. we will also explore related useful research tasks using extra collected information related to scientific work submissions in rr.