KEV i N : A Knowledge Enhanced Validity and Novelty Classifier for Arguments (2022)

Abstract: The ArgMining 2022 Shared Task is concerned with predicting the validity and novelty of an inference for a given premise and conclusion pair. We propose two feed-forward network based models (KEViN1 and KEViN2), which combine features generated from several pretrained transformers and the WikiData knowledge graph. The transformers are used to predict entailment and semantic similarity, while WikiData is used to provide a semantic measure between concepts in the premise-conclusion pair. Our proposed models show significant improvement over RoBERTa, with KEViN1 outperforming KEViN2 and obtaining second rank on both subtasks (A and B) of the ArgMining 2022 Shared Task.

## introduction a number of frameworks have been proposed to evaluate the quality of natural language arguments. many of these frameworks consider some notion of logical soundness (validity), @xcite . the argmining 2022 shared task also highlights the importance of novelty in measuring the usefulness of a conclusion in order to avoid redundant or non-informative conclusions. these metrics were more formally introduced in @xcite to assess the quality of arguments. in our work, we combine the power of pretrained language models with external knowledge sources to provide additional information for predictions @xcite . for this, we extract paths from wikidata @xcite that link the premise to the conclusion, and generate numerical features from these paths. having generated several sets of features using wikidata and pre-trained models, we proceed to use these features as inputs to a small feed-forward figure 1 : kevin 1 uses wikidata knowledge, and pretrained transformers to predict similarity and entailment prior to feeding the data into the neural network. network trained to predict validity and novelty. our results show a significant improvement from simply fine-tuning a pre-trained model on the task, and we identify that textual entailment serves as a strong indicator of argument validity, while a combination of textual entailment and knowledge graph distance serves to improve the model's ability to detect novelty foot_0 . we trained and tested two versions of our model on task a (binary classification). the first model, kevin 1 , predicts both validity and novelty using the same network (figure 1 ). the second model, kevin 2 , uses two separate networks, which were trained separately for each label, and then combines their predictions. both models show significant improvement over roberta with kevin 1 significantly outperforming kevin 2 . we additionally evaluated the kevin 1 model trained on task a on the testing set of task b, the corresponding details and results are given in the appendix. ## task and data description the argmining 2022 shared task consists of two subtasks: for a given textual premise, 1) classifying a conclusion as being valid/novel (task a) and 2) comparing conclusions in terms of validity/novelty (task b). for both tasks, the premise consists of multiple sentences while the conclusion is a single statement. validity requires that there exists a sound logical inference linking the premise to the conclusion; novelty, on the other hand, requires the conclusion to contain new information compared to the premise, and as such to be more than a simple paraphrase foot_1 . ## feature extraction in this section, we explain the process for extracting the features that were used as input to our feedforward network. ## preprocessing and training once the features were computed, we applied several preprocessing steps to improve results. given the distribution shift between the training and development data of task a, as shown in ## results and analysis for kevin 1 , our validation run identified irrelevancy, average kgp distance, te_p2c and te_c2p as the set of features leading to the best performance. we performed an ablation study to identify the relative contribution of each feature. this work identifies a strong similarity between argument validity and textual entailment which has been previously explored @xcite with mixed success for argument mining. likewise, the introduction of external kgs into the argument mining pipeline has been studied by @xcite ; @xcite ; @xcite . in the textual entailment literature, a substantial amount of work has shown the importance of external kgs in making accurate inferences over new domains @xcite . ## conclusion in this paper, we have shown how features can be extracted from knowledge graphs and pre-trained neural networks that are both relevant and complementary for the task of argument novelty and validity detection. we did this by demonstrating how a small neural network trained on these features outperforms fine-tuning with large transformers. we defined kg paths in terms of their semantic length and the corresponding kg distance features, which gave promising results and provides a basis for future work. for example, we would also like to consider semantic representations of paths, such as natural language representations, vector-based kg embedding approaches, and other kgs to improve the performance of the proposed model. in addition, it would be interesting to see if learning weights to predict the semantic length based on the relations in kg paths or if extending the graph containing the premise and conclusion concepts with semantic dependency relations would boost the performance.