Will It Blend? Mixing Training Paradigms & Prompting for Argument Quality Prediction (2022)

Abstract: This paper describes our contributions to the Shared Task of the 9th Workshop on Argument Mining (2022). Our approach uses Large Language Models for the task of Argument Quality Prediction. We perform prompt engineering using GPT-3, and also investigate the training paradigms multi-task learning, contrastive learning, and intermediate-task training. We find that a mixed prediction setup outperforms single models. Prompting GPT-3 works best for predicting argument validity, and argument novelty is best estimated by a model trained using all three training paradigms.

## introduction as debates are moving increasingly online, automatically processing and moderating arguments becomes essential to further fruitful discussions. the research field of automatic extraction, analysis, and relation detection of argument units is called argument mining (am, @xcite . the shared task of the 9th workshop on argument mining (2022) focuses on argument quality @xcite . argument quality can be broken down into multiple dimensions, each with its own purpose, or be extended to deliberative quality @xcite . the shared task includes two aspects of the logical argument quality dimension: validity and novelty. given a premise and a conclusion, a valid relationship indicates that sound logical inferences link the premise and conclusion. a novel relationship indicates that new information was introduced in the conclusion that was not present in the premise. prediction of an argument's validity and novelty can be either through binary classification (task a) or by explicit com-parison between two arguments (task b). we focus on task a. a system that is able to estimate validity and novelty could be a building block in am for online deliberation. for instance, in assisting humans to detect arguments in online deliberative discussions @xcite or presenting diverse viewpoints to users in a news recommendation system @xcite . we address the task of validity and novelty prediction through a variety of approaches ranging from prompting, contrastive learning, intermediate task training, and multi-task learning. our bestperforming approach is a mix of a gpt-3 model (through prompting) and a contrastively trained multi-task model that uses nli as an intermediate training task. this approach achieves a combined validity and novelty f1-score of 0.45. ## related work: paradigms & prompting given the two related argumentation tasks (novelty and validity), a multi-task learning (mtl) setup @xcite is a natural approach. multitask models use training signals across several tasks, and have been applied before in argumentrelated work with large language models (llms) @xcite @xcite . we use shared encoders followed by task-specific classification heads. the training of these encoders was influenced by the following two lines of work. first, intermediate task training @xcite fine-tunes a pretrained llm on an auxiliary task before moving on to the final task. this can aid classification performance, also in am @xcite . second, contrastive learning is shown to be a promising approach @xcite in a previous am shared task @xcite . contrastive learning is used to improve embeddings by forcing similar data points to be closer in space and dissimilar data points to be further away. such an approach may cause the encoder to learn dataset-specific features that help in downstream task performance. in addition to mtl, we look at prompt engineering for llms, which has shown remarkable progress in a large variety of tasks in combination with @xcite or without few-shot learning @xcite . for this task we draw inspiration from prop @xcite , an approach that ranked first in the "knowledge base construction from pre-trained language models" challenge at iswc 2022. 1 prop reports the highest performance with (1) larger llms, (2) shorter prompts, (3) diverse and complete examples in the prompt, (4) task-specific prompts. ## experiments and results we compare our approaches' performance on the test set with the shared task's metric (combined f1 of validity and novelty). additionally, we analyze our approaches' errors and their connection to labels, annotator confidence, and topic. ## conclusion we highlight two main conclusions.