Argument Pair Extraction via Attention-guided Multi-Layer Multi-Cross Encoding (2021)

Abstract: Argument pair extraction (APE) is a research task for extracting arguments from two passages and identifying potential argument pairs. Prior research work treats this task as a sequence labeling problem and a binary classification problem on two passages that are directly concatenated together, which has a limitation of not fully utilizing the unique characteristics and inherent relations of two different passages. This paper proposes a novel attention-guided multi-layer multi-cross encoding scheme to address the challenges. The new model processes two passages with two individual sequence encoders and updates their representations using each other’s representations through attention. In addition, the pair prediction part is formulated as a table-filling problem by updating the representations of two sequences’ Cartesian product. Furthermore, an auxiliary attention loss is introduced to guide each argument to align to its paired argument. An extensive set of experiments show that the new model significantly improves the APE performance over several alternatives.

## introduction mining argumentation structures within a corpus is a crucial task in argument mining research field @xcite . there are usually two main components in learning natural language argument structures: (1) detecting argumentative units, (2) predicting relations between the identified arguments. it has been widely studied by natural language processing (nlp) researchers @xcite and applied to domains such as: web debating platforms @xcite @xcite , persuasive essays @xcite , social media @xcite , etc. unlike traditional argument extraction tasks that are mainly from monologues, @xcite propose a new task -argument pair extraction (ape) from two passages in a new domain, namely peer review process, focusing on exploiting the interactions between reviewer comments and author rebuttals. as shown in figure 1 , ape task aims to extract the argument pairs from two passages. specific suggestions, questions or challenges in reviews are considered as review arguments. response sentences that answer or explain the specific review argument are its paired rebuttal arguments. for example in the pink area, the reviewer points out the lack of literature review in submission (i.e., review sentences 11-12). as a response, the authors argue that they select the literature based on the special focus of their work (i.e., rebuttal sentence 6-7). similar to the two components in the traditional argumentation structure mining, the ape task can be divided into two subtasks: (1) extracting the review and rebuttal arguments from two passages, (2) predicting if an extracted review argument and a rebuttal argument form an argument pair. the first subtask can be cast as a sequence labeling problem and the second one can be cast as a binary classification problem. one straightforward approach is to couple the two subtasks in a pipeline. however, such a pipeline approach learns two subtasks independently without sharing ample information. to address this limitation, the pioneering work @xcite ) employs a multi-task learning framework to train two subtasks simultaneously. however, there are several shortcomings in the multi-task model. first, the review passage and its rebuttal passage are concatenated as a single passage to perform the argument extraction subtask with sequence labeling. it is obvious to see from second, the subtask coordination capability of their multi-task framework is weak as two subtasks only coordinate with each other via the shared feature encoders, i.e., the sentence encoder for the sequence of word tokens and the passage encoder for the concatenation of sentences. thus, the shared information between two subtasks is only learned implicitly. to overcome this limitation, we propose an attention-guided multi-layer multi-cross (mlmc) encoding mechanism. inspired by the table-filling approach @xcite , we form a table that represents features for the cartesian product of review and rebuttal sequences by utilizing both of their embeddings, as shown in the right portion of figure 1 . the table representations will be updated with the incorporation of the two sequence representations, and in return, it will also help to update the mutual attention mentioned above. it is named as multi-cross encoder because these three encoding components (i.e., one table and two sequences) interact with each other explicitly and extensively. by stacking multiple encoder layers, the two subtasks can further benefit each other. in addition, we also design an auxiliary attention loss to guide each argument to refer to its paired arguments. this additional loss not only enhances the model performance, but also significantly improves the attention interpretability. to summarize, the contributions of this paper are three-fold. firstly, we apply the table-filling approach to model the sentence-level correlation between two passages with multiple sentences for the first time. secondly, on the model side, we propose an mlmc encoder to explicitly learn the useful shared information in the two passages. furthermore, we introduce an auxiliary attention loss, which is able to further improve the efficacy of the mutual attentions. thirdly, we evaluate our model on the benchmark dataset @xcite , and the results show that our model achieves a new state-of-the-art performance on the ape task. ## related work argument mining has wide applications in educational domain, including persuasive essays @xcite , scientific articles @xcite , writing assistance @xcite , essay scoring @xcite , peer reviews @xcite , etc. unlike previous works, @xcite introduce a new task named ape in the domain of peer review and rebuttal, which intends to extract the argument pairs from two passages simultaneously. table-filling approaches @xcite @xcite have been proposed to work towards the joint task of name entity recognition (ner) and relation extraction (re). in their work, the diagonal entries of the table show the words' entity types and the off-diagonal entries show the relation types with other words. more recently, there are more research works to propose various table-filling models on different tasks. @xcite propose to learn two separate encoders (a table encoder and a sequence encoder) by interacting with each other for joint ner and re task. @xcite propose a grid tagging scheme to address the aspectoriented fine-grained opinion extraction task. compared to our model, one major difference is the table shape. in their tables, the row and column represent the same sequence, and thus in square shape. in our model, the table is in a rectangle shape where the row and column represent two different sequences with different lengths. another clear difference is that each entry in their table is for word-pair relation, whereas each entry in our table captures sentence-pair relation. as we can see from figure 1 , the review/rebuttal sequence consists of a list of sentences. thus, it requires extra effort to learn comprehensive sentence representations. ## task formulation in this paper, we tackle the ape task, which aims to study the internal structure and relations between two passages, e.g., review and rebuttal passages. for example, as shown in figure 1 , given a pair of review passage s @xmath0,1 , • • • , s rv,12 ] (in the red box) and rebuttal passage s @xmath1,1 , • • • , s rb,7 ] (in the orange box), we intend to automatically extract all argument pairs between them. first, for the argument mining subtask, we cast it as a sentence-level sequence labeling problem follow-ing the work @xcite using the standard bio scheme @xcite . this subtask segments the argumentative units (highlighted in blue/pink) from nonargumentative units (highlighted in grey) for each passage. the label sequences for the review passage and the rebuttal passage are shown in the right portion of figure 1 . second, the sentence pairing subtask predicts whether the two sentences belong to one argument pair. here, we formulate it as a table-filling problem following the work @xcite . take the 8th review sentence s rv,8 in the first review argument as an example, the rebuttal argument sentences {s rb,2 , s rb,3 , s rb,4 , s rb,5 } forming sentence pairs with it are filled with green, as shown in the table. with the collaboration of these two subtasks, we can perform the overall argument pair extraction task. in this case, two argument pairs (highlighted in blue/pink from two passages) are extracted, which correspond to the two green rectangles shown in the table. ## model figure 2 shows our proposed attention-guided multi-layer multi-cross (mlmc) encoding based model. the model mainly consists of three parts: a sentence embedder, an n-layer multi-cross encoder, and a predictor. the review sentences and rebuttal sentences first go through the sentence embedder separately to obtain their sentence embeddings respectively. we then utilize the representations from review and rebuttal sequences to form a table as shown earlier in figure 1 . next, the representations of the table and two sequences are updated through n multi-cross encoder layers. finally, the model predicts the review and rebuttal arguments through a conditional random field (crf) @xcite layer based on two sequence representations, and extracts the pairing information through a multi-layer perceptron (mlp) based on the table representations. ## conclusions in this paper, we adopt the table-filling approach for modeling the sentence-level correlation between two passages, and propose the attention-guided multi-layer multi-cross (mlmc) encoding scheme for the argument pair extraction (ape) task. our model can better capture the internal relations be-tween a review and its rebuttal with two sequence encoders and a table encoder via mutual attention mechanism. we also introduce an auxiliary attention loss to further improve the efficacy of the mutual attentions. extensive experiments on the benchmark dataset demonstrate the effectiveness of our model architecture, which is potentially beneficial for other nlp tasks.