Argument Novelty and Validity Assessment via Multitask and Transfer Learning (2022)

Abstract: An argument is a constellation of premises reasoning towards a certain conclusion. The automatic generation of conclusions is becoming a very prominent task, raising the need for automatic measures to assess the quality of these generated conclusions. The SharedTask at the 9th Workshop on Argument Mining proposes a new task to assess the novelty and validity of a conclusion given a set of premises. In this paper, we present a multitask learning approach that transfers the knowledge learned from the natural language inference task to the tasks at hand. Evaluation results indicate the importance of both knowledge transfer and joint learning, placing our approach in the fifth place with strong results compared to baselines.

## introduction conclusions are essential to understanding the reasoning behind their arguments. in daily life argumentation, argument conclusions are often left implicit @xcite because they are easy to infer or for rhetorical reasons. while it is easy for humans to infer these conclusions, machines struggle with such a task. this phenomenon motivated a line of computational argumentation research to study the task of automatic generation of conclusions @xcite . evaluating these approaches using traditional text generation measures like bleu or rouge is not enough since multiple conclusions can be considered valid for a given argument. additionally, one might desire specific criteria in a generated conclusion, like being informative @xcite . in this regard, the sharedtask at the 9th workshop on argument mining proposed two quality dimensions of argument conclusions to be assessed. the first is validity, defined as whether a given conclusion can be logically inferred from its premises. the second is novelty, assessing whether the conclusion goes beyond what is mentioned in the validity classification us offshore oil drilling : these large ships release significant pollution into the oceans, and carry some risk of hitting the shore, and causing a spill. </@xmath0 figure 1: our proposed model, which jointly models the validity and novelty assessment tasks, starts from a transformer-based model pre-trained on the natural language inference (nli) task. first, we pass the input through the roberta encoder. then, the last hidden state of the encoder is projected into a probability distribution representing the nli labels. each classification head (novelty and validity) then learns to map this distribution into the corresponding labels. we address the novelty and validity assessment tasks via a multitask learning approach, employing already acquired knowledge from the natural language inference task. in particular, the two assessed quality dimensions are orthogonal. that is, conclusions that can be easily inferred from their premises and hence valid are less likely to be novel. similarly, the more novel a conclusion is, the harder to judge its validity. accordingly, we believe that jointly modeling the two assessment tasks allows the model to exploit such dynamics. moreover, the natural language inference task (nli) is very simi-lar and is widely studied @xcite . one can understand the entailment attribute between two sentences in the nli task as a validity criterion. hence, we start from a transformer-based model fine-tuned on the nli task. as shown in figure 1 , this model consists of a transformer-based encoder and a classification head that predicts one of three labels, entailment, contradiction, neutral. we stack two classification heads on top of the model, one to predict novelty and the other for validity. we evaluate our approach against the basic roberta model trained on each task independently in our experiments. results show the gain achieved from both the knowledge acquired from the nli task as well as the joint learning of the two tasks. first, utilizing knowledge from the nli task boosts the average f1-score from 0.09 to 0.15. secondly, the joint learning of the two tasks further raises the average f1-score up to 0.42, placing our approach in the fifth place with strong competitive performance. foot_0 2 related work conclusion inference is the task of generating a natural language conclusion given a set of premises. the generation of these conclusions is important for ai algorithms to understand the reasoning behind arguments. hence, several works in computational argumentation addressed this task. @xcite reconstructed implicit conclusion targets from premises using triplet neural networks. @xcite studied the effectiveness of several transformer-based models on the conclusion generation across various corpora and evaluated the informativeness criteria of conclusions. @xcite automatically generated conclusions to then use them for argument quality assessment. @xcite worked on generating perspectives (conclusion) for news articles. @xcite fine-tuned language models to generate implicit knowledge in sentences. in this work, the proposed task and approaches aim to study the quality of automatically generated conclusions along the validity and novelty dimensions. recent advances in natural language processing (nlp) have been driven by transfer learning, where knowledge on one task is used to learn another potentially relevant task. indeed, it has been shown that language models trained on big corpora can excel in transferring such knowledge into downstream tasks in a zero-shot setting @xcite . our proposed method uses knowledge learned the natural language inference (nli) task @xcite to solve the novelty and validity assessment tasks. another promising learning paradigm is multitask learning @xcite , in which two or more relevant tasks are learned in the same neural model, either in a soft or hard parameter sharing setting. our approach models the validity and novelty tasks jointly in one model with hard parameter sharing. ## task and data in the sharedtask at the 9th workshop on argument mining, the organizers defined the validity and novelty criteria of argument conclusions as follows: foot_1 according to these definitions, the organizers proposed two settings for this sharedtask. the first is, given a set of premises in natural language text and a corresponding conclusion, predict two scores that reflect the conclusion's novelty and validity (subtask a). in the second subtask, two conclusions are provided, and the task is to rank them according to their novelty and validity (subtask b). this paper tackles subtask a, which is the binary classification of validity and novelty dimensions. the dataset provided by the organizers consists of premises and conclusions, which they manually annotated for validity and novelty dimensions. additionally, the organizers include the topic of the debate and the confidence scores for the two labels. the data also contained borderline cases for both target dimensions, which are considered to be somewhat novel or valid. we excluded those examples from the training and validation sets, as suggested by the organizers. we ended up with 721 training and 199 development examples for validity and 718 training and 200 development examples for novelty. the test set has a size of 520 instances. ## approach as mentioned, our approach to the subtask a is to jointly learn the two assessment tasks (novelty and validity), starting from knowledge acquired by a model trained on the nli task @xcite . the motivation for our choice is two folds. on the one hand, we argue that the novelty and validity dimensions correlate such that conclusions that are easily inferred to be valid are likely not that novel. similarly, the more novel a conclusion is, the harder to judge its validity. on the other hand, we see similarities to the natural language inference task. if an nli model deemed the conclusion to be entailed from its premises, then the conclusion is likely valid but probably not novel. in particular, we start from a transformer-based model fine-tuned on the nli task. as shown in figure 1 , this model consists of a transformer-based encoder and a classification head that predicts one of three labels, entailment, contradiction, neutral. the input to our model is a concatenation of the topic, premise, and conclusion. we pass the input through the roberta encoder to obtain the final hidden state, which is passed through the classification layer to obtain a probability distribution over the three nli labels. we stack two classification heads on top of the model to project this distribution into the corresponding novelty and validity labels. during training, one can compute an average error with respect to the two tasks at each optimization step or consider the error subject to one task at a time. for simplicity, we chose the second option since the framework we build upon supports only this option (details in sections 5). although the weights are not updated according to an average loss of the two tasks, the overall training will drive the weights into an area optimal for both tasks. ## experiments in our experiments, we use the roberta model @xcite fine-tuned on the multi-genre natural language inference dataset (mnli) made public by @xcite . for each task, we train a model considering it the main task and the other as an auxiliary one with a loss discounted by a factor of α. we explored a range of α values and chose the ones that lead to the best f1-score on the validation set, that is, 0.9 and 0.7 for novelty and validity, respectively. additionally, we explored different learning rates for each of the models independently and chose a learning rate of 2e -5 and 5e -6 for the novelty and validity models, respectively. we train both models for ten epochs with a batch size of 8. we compare our approach against the roberta model without nli fine-tuning and once with nli fine-tuning. both trained independently on each task. as mentioned in section 3, the training data is imbalanced along the novelty label. to address this problem, we perform oversampling in which we randomly replicate instances of the class novel to reach a balanced situation. we then train our model and the baselines on it. our model is built on top of the multitask learning framework made publicly available under https://multi-task-nlp. readthedocs.io/en/latest/ . from 0.28 to 0.52. moreover, modeling novelty and validity tasks jointly boost the performance to reach 0.71 and 0.60 f1-score on the validity and novelty tasks, respectively. the combined f1-score recognizes instances as correctly predicted only if validity and novelty are both correctly predicted. among evaluated baselines in this paper, our model achieves the best combined f1-score of 0.42. from ## conclusion in this paper we described our approach proposed for the sharedtask at the 9th workshop on argument mining for assessing the validity and novelty of argument conclusions. our approach jointly models the two binary tasks of novelty and validity making use of knowledge acquired from the natural language inference task. experimental results, shows the gain achieved from both transferring knowledge from the nli, as well as the joint modeling of the two tasks.