Probing Structural Knowledge from Pre-trained Language Model for Argumentation Relation Classification (2022)

Abstract: Extracting fine-grained structural information between argumentation component (AC) pairs is essential for argumentation relation classification (ARC). However, most previous studies attempt to model the relationship between AC pairs using AC level similarity or semantically relevant features. They ignore the complex interaction between AC pairs and cannot effectively reason the argumentation relation deeply.Therefore, in this paper, we propose a novel dual prior graph neural network (DPGNN) to jointly explore the probing knowledge derived from pre-trained language models (PLMs) and the syntactical information for comprehensively modeling the relationship between AC pairs. Specifically, we construct a probing graph by using probing knowledge derived from PLMs to recognize and align the relational information within and across the argumentation components. In addition, we propose a mutual dependency graph for the AC pair to reason the fine-grained syntactic structural information, in which the syntactical correlation between words is set by the dependency information within AC and mutual attention mechanism across ACs. The knowledge learned from the probing graph and the dependency graph are combined to comprehensively capture the aligned relationships of AC pairs for improving the results of ARC. Experimental results on three public datasets show that DPGNN outperforms the state-of-the-art baselines by a noticeable margin.

## introduction argumentation relation classification (arc) is the most challenging subtask of argumentation mining, which requires the model to understand complex linguistic interactions between argumentation components @xcite . the goal of arc is to identify the argumentation relation (i.e., support or attack) between argumentation components (ac) pairs. as shown in figure 1 , there is an example with support relation and an example with attack relation, where ac2 supports ac1 and ac4 attacks ac3. arc needs the model to effectively explore structural knowledge within and cross acs so as to better infer the argumentation relation between ac pairs. intuitively, the semantic structure can capture the correlations among semantically similar words, while the syntactic structure can capture the syntactical constraint (e.g., dependency information) for syntactically relevant words. taking figure 1 as an example, the words "affirmative action" and "is good" in ac1 can be aligned with "diversity" and "improves" in ac2 respectively for identifying the support relation. however, most previous efforts @xcite @xcite merely focus on the ac-level similarity between ac pairs and result in sub-optimal performance. although recent works @xcite model the fine-grained semantically relevant features (such as words) between ac pair by introducing external knowledge, they ignore the complex interactions of ac pair and cannot effectively reason the argumentation relation deeply. for instance, in figure 1 , previous works only empha-size the superficial similarity between individual words "better" and "priority" and fail to capture the overall opposition relation between ac3 and ac4 since "electric cars" is the object in ac3 but the subject in ac4. to our best knowledge, the structural knowledge within or across acs has not been simultaneously investigated for the argumentation relation classification. in this paper, we propose a dual prior graph neural network (dpgnn) to leverage two prior knowledge, i.e., dependency information and probing knowledge from pre-trained language models (plms), by constructing dual graph modules for arc. by combining the information from the two graph modules, dpgnn can more accurately capture the fine-grained relations between ac pairs. specifically, we construct the mutual dependency graph from two perspectives (i.e., intra-ac perspective and inter-ac perspective) to gain the syntactical structure information with dependency parsing and attention mechanisms. the intra-ac graph constructs the syntactical structure within acs in the mutual graph, and the inter-ac graph aligns and builds the structure between ac pair to explore the argumentation relation. despite the effectiveness of the learned dependency information, it is difficult to recognize semantically relevant words such as synonyms and antonyms by only relying on the dependency information. to complement the dependency information for arc, we probe the relational knowledge from plms, such as bert (kenton and toutanova, 2019) and roberta @xcite , which contain rich semantic knowledge for arc, such as the counterpart relationship between "public policy" and "group decision-making" as shown in figure 1 . concretely, we first probe token representations and attention matrices from plms, where the token representations form the nodes of the probing graph and the attention matrices form the edges between nodes. in particular, to obtain useful probing knowledge with respect to arc, we propose three different levels of probes 1 (i.e., word-, acand pair-level) with adaptive attention mechanisms for probing token representations and attention matrices. the key idea behind this probing technique is motivated by the observation that the knowledge stored in plms contains rich linguistic and relational knowledge (petroni et al., 2019; zhong et al., 1 a probe is a simple neural network that develops the features (i.e. hidden states and attention weights) from plm for specific task @xcite 2021) about words (e.g., synonyms and antonyms) in plms @xcite . we disentangle each probed attention matrix into four attention sub-matrices according to the span of each ac within ac pair to construct the probing graph from intra-ac and inter-ac perspectives, respectively. finally, we combine the graph representations from the dependency graph and the probing graph with a biaffine function @xcite for argumentation relation prediction. our main contributions are three-fold. ( 1 ) we propose a dual prior graph neural network for arc, which jointly explores the probed knowledge derived from plms and the syntactical information to comprehensively model the relationship between ac pairs. (2) we probe rich relation knowledge from plms in terms of ac-pair level, which elicit relations to robustly capture the semantic correspondences between acs and ac pairs. (3) we conduct extensive experiments on three benchmark arc datasets. experimental results show that our method significantly outperforms previous methods and achieves new state-of-the-art results. ## methodology task definition following previous work @xcite , we assume an ac pair (p, q) are given in an argumentative text, where the argumentation components @xmath0, p 2 , . . . , p m ) and @xmath1, q 2 , . . . , q n ) consist of m and n tokens, respectively. the goal of arc is to predict the relation type y (p,q) (i.e., support or attack). model overview figure 2 illustrates the architecture of our dpgnn model. our method leverages two complementary structural knowledge to construct dual graphs for learning the alignment of fine-grained structures within acs and between ac pairs for arc. concretely, we propose three probes to elicit knowledge from hidden states and attention matrices in bert. the probed hidden states and attention matrices are employed to construct a probing graph for reasoning the relation between the ac pair. in addition, we develop the dependency graph to gain the syntactical structure information with dependency parsing and attention mechanisms. finally, a biaffine module is devised to combine the probing graph and the mutual dependency graph for improving the performance of arc. ## conclusion in this paper, we proposed a graph-based model dpgnn with two prior knowledge, i.e., probing knowledge elicited from plm and syntactical dependency information, to model the relational and syntactic structures within acs and between ac pairs for arc. to effectively capture the useful probing knowledge from bert, we propose three probes to elicit word-, ac-and pair-level knowledge. in addition, dpgnn integrated the probing graph with decoupled probing attention matrices and the mutual dependency graph with syntactic dependency information to make our model more effective to utilize the heterogeneous structure within acs and between ac pairs. experimental results on three benchmark datasets demonstrated that dpgnn outperformed the strong baselines.